:W&Bá¾Å‰ˆ‹M mb2gxnxyª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0á7– ú‚" azp5f8zfycu5 mb2gxnxyŠÖ mb2gxnxyjuju11-uc-san-diegoSPAtop"™ _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.12687233479240043 epochs‚10  hidden_dim‚64 ' learning_rate‚0.0008023186710573254  num_attention_heads‚8  num_branch_encoder_layers‚7  num_embedding_layers‚14  num_encoder_layers‚2Bjjtttt_vanilla_mb2gxnxyb88oquqqwj)jj-spatop-sweepagent-tttt-vanilla-2-spfhcŠ ¤äÔðèÝšš3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64‹RP‰7 mb2gxnxy&  _runtime‚0  _wandb.runtime‚0ÜwëjÒä -Linux-6.8.0-136-generic-x86_64-with-glibc2.36CPython 3.12.10 ¤äÔà×½*-of*=/data/spatop/options_files/sweep_tttt_jj/config_mb2gxnxy.json*-l* /scratch/logs*-n*jjtttt_vanilla_mb2gxnxy2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb)jj-spatop-sweepagent-tttt-vanilla-2-spfhcr/opt/conda/bin/pythonº vjv5tdo47wctx057vya9jqxikfejc7nbú·9BÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€à‚´¢"ª€àäÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-1512559d-7724-700d-39a3-dab660c9a52eÊ13.3º vjv5tdo47wctx057vya9jqxikfejc7nbµª½Ç2  requirements.txt‚M¾W mb2gxnxyZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64|gð±Y mb2gxnxyZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64˜‘Û*  mb2gxnxyb trainer/global_step2sÞI\  mb2gxnxyZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64wnÖ/  mb2gxnxyb*"trainer/global_step2n>e·K  mb2gxnxyj: ‘¤äÔ¸ªíœ*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] æéB'“  mb2gxnxyj  ‘¤äÔðˆýšî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 166 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 13.6 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ àÞl\ mb2gxnxyjÊ ‘¤äÔð—õ›·Trainable params: 14.0 M Non-trainable params: 86 Total params: 14.0 M Total estimated model params size (MB): 55.828 Modules in train mode: 5777 Modules in eval mode: 0 Total FLOPs: 0 ¾œO×q mb2gxnxyj` ‘¤äÔ ´œPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). EÖ:{ mb2gxnxyjj ‘¤äÔØ› œZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). zQ¯šy mb2gxnxyjh ‘¤äÔðé­œXwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). ’©ƒG€ mb2gxnxyjo ‘¤äÔ蘻œ_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). à‘Ûz mb2gxnxyji ‘¤äÔ°éÂœYwandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). :FeŸt mb2gxnxyjc ‘¤äÔ€ÞËœSwandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). §%5[q mb2gxnxyj` ‘¤äÔøžÓœPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). œ„Äýt mb2gxnxyjc ‘¤äÔüÛœSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). ¡Qrûn mb2gxnxyj] ‘¤äÔ°†ãœMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). ß9m mb2gxnxyj\ ‘¤äÔÈÚîœLwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). àß â_ mb2gxnxy*Í  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚0  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"—/{Ïë mb2gxnxyjÙ ‘¤äÔÀËÔ¢È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. _M‘pN mb2gxnxyj¼ ‘¤äÔ˜Ø⢫/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. .› ë mb2gxnxyjÙ ”¤äÔ¨ö“˜È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. êúP mb2gxnxyj¾ ”¤äÔÈ壘­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ‘lå‚(:…  ¤äÔä¢x gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.662185668945312) gpu.0.memoryAllocatedBytes‚ 7440760832 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FÄÐÓ÷‚(:í  ¤äÔ§±~ disk.rbd2.out‚ 1.51171875 disk.sda.out‚0.625 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0$ memory_percent‚97.05333709716797" proc.memory.rssMB‚ 2755.3671875 network.sent‚35330% disk./.usageGB‚1096.6014213562012 disk.rbd4.in‚0 disk.drbd1777.out‚0% proc.memory.availableMB‚ 482.78125 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4 disk.rbd1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.817426681518555 disk.sda1.out‚0 disk.rbd1.out‚0 disk.dm-0.in‚4 disk.dm-0.out‚ 0.51025390625 disk.dm-1.out‚0.625 network.recv‚42587) disk./.usagePercent‚73.59384149709274 disk.rbd3.out‚0 disk.drbd1777.in‚0$ disk.nvme0c0n1.out‚ 0.51025390625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.in‚0 disk.sda3.out‚0.625 disk.nvme0n1.in‚4" disk.nvme0n1.out‚ 0.51025390625 disk.dm-1.in‚0 cpu‚24.91863539924101 proc.cpu.threads‚17 disk.sda2.out‚0àò.ë ‚(:… ¯¤äÔ¥î‰x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.662185668945312) gpu.0.memoryAllocatedBytes‚ 7440760832 gpu.0.temp‚31.0 gpu.0.powerWatts‚123.941' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.ב°!‚(:ý ¯¤äÔ¸µ®z disk.drbd1777.out‚0 proc.memory.rssMB‚ 2755.40625 cpu‚24.957262186295893 disk.sda2.in‚0 disk.rbd4.in‚0 disk.dm-0.out‚ 0.9423828125 disk.rbd2.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 0.9423828125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0! disk.nvme0n1.out‚ 0.9423828125# memory_percent‚97.0590591430664 proc.cpu.threads‚17' disk./.usagePercent‚73.593862233703 disk.rbd1.out‚ 0.06298828125 disk.sda2.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4 disk.rbd3.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0* proc.memory.percent‚16.817665100097656 disk.rbd3.out‚0 disk.dm-0.in‚4 disk.rbd2.out‚ 3.1953125 disk.drbd1777.in‚0% proc.memory.availableMB‚ 481.84375 disk.sda.out‚ 1.0859375 disk.rbd1.in‚0 disk.nvme0n1.in‚4 disk.dm-1.out‚ 1.0859375 network.sent‚52937 network.recv‚62043% disk./.usageGB‚1096.6017303466797 disk.sda3.out‚ 1.0859375ªÏ©¥"‚(:ƒ ¾¤äԻУx gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.662185668945312) gpu.0.memoryAllocatedBytes‚ 7440760832 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.26' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.652 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w]Ý#‚(:ˆ ¾¤äÔž€—{ disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 1.6953125 proc.cpu.threads‚17 network.sent‚66574 network.recv‚80894) disk./.usagePercent‚73.59385148138658 disk.sda.out‚ 1.6953125* proc.memory.percent‚16.817784309387207 disk.dm-0.out‚ 2.55810546875 disk.rbd2.out‚ 3.1953125 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 2.55810546875 disk.rbd3.out‚0" disk.nvme0n1.out‚ 2.55810546875 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 481.2890625% disk./.usageGB‚1096.6015701293945 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda1.in‚0$ memory_percent‚97.06244468688965 disk.dm-0.in‚4 disk.nvme0c0n1.in‚4# proc.memory.rssMB‚ 2755.42578125 cpu‚24.976141299657435 disk.rbd2.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚4 disk.sda2.in‚0 disk.dm-1.out‚ 1.6953125 disk.rbd1.out‚ 0.062988281253c€Ï3$b/ lr-AdamW/pg1"trainer/global_step2:JP˜‰7&3%b/ lr-AdamW/pg2"trainer/global_step2:JPð]-&) _wandbruntime‚49  _runtime‚492©(ºÂ'½ _step‚0 ' lr-AdamW/pg1‚3.2517464749221624e-05 ' lr-AdamW/pg2‚3.2517464749221624e-05 trainer/global_step‚49 % _timestamp‚1.7884165785030625e+09 _runtime‚ 49.253069986Ö¤%ÂÄ(¿ ' lr-AdamW/pg1‚3.2517464749221624e-05 ' lr-AdamW/pg2‚3.2517464749221624e-05 trainer/global_step‚49 % _timestamp‚1.7884165785030625e+09 _runtime‚ 49.253069986 _step‚02ç$g)bc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPUõ®r*bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPà„ñ½f+bb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JP)¼‚îp,bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPú{+Ëg-bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPÏàÂl.bh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JP^ÂëÓp/bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPu¹Òf0bb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPÒKým1bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP|W…j2bf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP¸!!d3b` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JPq®h4bd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JP9}åh5bd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP”–RGf6bb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPR4§8p7bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPž°Ij8bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP¼=zWg9bc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP|¶iÄl:bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP°ÈŠët;bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPlÚy¦jbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPŒß¡Ý_?b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JPK¹Làg@bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP—uíjAbf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPx9ÄiBbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP¶\ÂïkCbg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JP­Û'ÔpDbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP㾎élEbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP7™moFbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPÙÔõíeGba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP_^ŠÛmHbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP*¸ çvIbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP%käàqJbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP&—!çqKbm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP¬ÂçófLbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPÊíê¯hMbd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPê«®ÒiNbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPZi/fObb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JP™òDwmPbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPNž§çmQbi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JPÅVènRbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP=öXÆiSbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP›ÚrßWTbS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JP…)_Ub[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPrr  iVbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPCDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP¦)LmiXbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP¶TvÇoYbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP;>kZbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPDk÷Vm[bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP2-hÁb\b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPø n]bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPè~Ür^bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPÿÛÀf_bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JP¥?ue`ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPÊ”æ9gabc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPâ‡ú®ebba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP‘qÉrcbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP¼Ëþ£]dbY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPp$c]^ebZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPÇ~wsfbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP( \ãggbc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPýVÛ3rhbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP®ÿÒhibd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP}»j‚njbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPž0”Ïnkbj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP¼.Úfalb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP@–+imbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP¥ä*­qnbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPîõ#hiobe BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JPˆ+hŽipbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPr…@pqbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP. grbc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JP–¢¢vsbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPìpÁatb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPÑ Þtubp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPMarkvbg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPj˜¼Pnwbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPªËÕ‘hxbd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPOœIcyb_ DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JP÷Çq€bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP­ Ùû`b[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPœW÷m‚bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPûñ¨Šlƒbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPt"d·t„bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP~ìAïh…bc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP²ím†bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP‰¯ñm‡bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP‰®ùpˆbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPØrˆÜl‰bg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JP{’›eŠb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JP±"Úl‹bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPGŽoŒbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP2¸L«obj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPŸi=•oŽbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPÊ+Â^_bZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JP ‹yWibd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP/ûÖ¹k‘bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPÿèOÝa’b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP-jl%p“bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP!bõo”bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPø£…l•bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP¤µå5k–bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JPT{ÛXy—bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPV ¤ør˜bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPhÂœvt™bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP²|õ gšbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPR^"ýo›bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPð’0Ütœbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP°DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2167594 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 1933944256512 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541124608 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 91020142592 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2191768 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1919095362814 _step‚1 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2024982402048 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 90792706048 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚853580 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚261 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541124608 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚323935 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 % _timestamp‚1.7884165785038304e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29897 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2024736962560 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚898103 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1828326333440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 2073001539584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 280350208 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6891241472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1933962259456 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1188 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269588480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2166052962816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚261 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1206 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚261 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29668 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2024982402048 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 90528348414 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6891241472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2024736962560 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚324558 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 90751990014 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 547356672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0q“žˆ'¦‚O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541124608 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 91020142592 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2191768 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1919095362814 _step‚1 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2024982402048 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 90792706048 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚853580 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚261 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541124608 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚323935 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 % _timestamp‚1.7884165785038304e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29897 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2024736962560 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚898103 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1828326333440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 2073001539584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 280350208 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6891241472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1933962259456 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1188 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269588480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2166052962816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚261 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1206 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚261 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29668 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2024982402048 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 90528348414 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6891241472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2024736962560 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚324558 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 90751990014 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 547356672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚529020 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚898105 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29668 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 547356672 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚852955 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2072977599488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287117 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2167594 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 1933944256512_ø-]§bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPÔ>5Ûq¨bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPlBç˜u©bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP³ô³lªbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP¹µ3×d«b_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP˜þ“r°bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP!°–f±ba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPÅJÊf²ba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP/¼è¾b³b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP3cOü`´b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JP.yggµbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP»­¾m¶bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPO5g·bb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPçAq\]¸bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPeh¹bc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPMÓnXqºbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPªÎ%k»bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JP(bRìd¼b_ DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP7 'QiÒbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JP¾K„SjÓbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPçÇ÷fÔba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP=—VˆkÕbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP ñ_ÖbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP¯*i+h×bc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPÖYjØbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP½%ÓgÙbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JP•ª5cÚb^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JPÓÿMkÛbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPgpÜbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP;êÙiÝbd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JP¼&–bnÞbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPÊI›rßbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP£–Ãiàbd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPžTÁ(hábc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JP#¨¨uâbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPQs`kãbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP.Âîläbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPûRŽnåbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPÁ¾b]bæb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JP•/ã`çb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPû[}ièbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JP„‹oébj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP9O]}oêbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPí·jÔfëba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP™­ÙMlìbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPÜkMLdíb_ DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPb(Ylbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPórIobj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPC‚?Ëm‘bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP³Ú».g’bb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP)²â r“bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP¹¢ïµq”bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPDž#o•bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP÷9¼¬w–br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPr@»e—b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JP‚zhÈe˜b` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JP§‚¢èf™ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP˳ ‚cšb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPJò0¬j›be BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JP«D~(mœbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPUQ£Seb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP¹l<)sžbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPÏ(C¦^ŸbY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPóºúYj be BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JP4K‰Ôp¡bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPs.¨¡.¢) _wandbruntime‚50  _runtime‚50¥—|%䣉M VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 92360748014 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2210864 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚915822 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 297709568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2114147295744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6891241472 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚330011 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1956989539310 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 92584389822 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 2114171235840 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚261 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 _step‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚539375 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1295042 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 94658173952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚330633 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273769472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚869386 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 280350208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 D>Devic?ÏêN«eStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2235064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 547356672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 321957888 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1319240 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6891241472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 92630400512 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚261 trainer/global_step‚49 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 1972282200576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1864628791296 _runtime‚ 50.108992773 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 1972300203520 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1295042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1957230220478 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 547356672 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 92630400512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1972300203520 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 92857850368 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2235064 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2208805469696 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 _step‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚539375 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1295042 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 94658173952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚330633 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273769472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚869386 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 280350208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2235064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 547356672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 321957888 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1319240 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6891241472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 92630400512 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚261 trainer/global_step‚49 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 1972282200576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1864628791296 _runtime‚ 50.108992773 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 1972300203520 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1295042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1957230220478 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 547356672 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 92630400512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1972300203520 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 92857850368 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2235064 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2208805469696 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6891241472 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚330011 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1956989539310 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 92584389822 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 2114171235840 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚261¶¶ròA¥b< loss/FBt2/assignment_loss"trainer/global_step2:JP½}¥B¦b= loss/SRqqt2/detection_loss"trainer/global_step2:JP,òX³A§b< loss/FBt3/assignment_loss"trainer/global_step2:JPª‡B¨b= loss/SRqqt1/detection_loss"trainer/global_step2:JPkµñ¯A©b< loss/FRt3/assignment_loss"trainer/global_step2:JPÔwÝùCªb> loss/SRqqt2/assignment_loss"trainer/global_step2:JP++@«b; loss/FBt2/detection_loss"trainer/global_step2:JPm .Ì@¬b; loss/FBt3/detection_loss"trainer/global_step2:JP ‡zA­b< loss/FRt2/assignment_loss"trainer/global_step2:JP=Ûä÷@®b; loss/FRt2/detection_loss"trainer/global_step2:JP¡¾ÄÂA¯b< loss/FBt1/assignment_loss"trainer/global_step2:JPx¯e¹@°b; loss/FRt4/detection_loss"trainer/global_step2:JP!È}B±b= loss/SRqqt3/detection_loss"trainer/global_step2:JP§â\Ò@²b; loss/FBt1/detection_loss"trainer/global_step2:JPf$|A³b< loss/FRt4/assignment_loss"trainer/global_step2:JP>aÿ_C´b> loss/SRqqt4/assignment_loss"trainer/global_step2:JPYÂAµb< loss/FBt4/assignment_loss"trainer/global_step2:JPÅP×SB¶b= loss/SRqqt4/detection_loss"trainer/global_step2:JPߘ-·b( epoch"trainer/global_step2:JP@Y¸mA¸b< loss/FRt1/assignment_loss"trainer/global_step2:JPuÞ@¹b; loss/FRt1/detection_loss"trainer/global_step2:JPçaê@ºb; loss/FRt3/detection_loss"trainer/global_step2:JPb ô7»b2 loss/total_loss"trainer/global_step2:JPówG„C¼b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPOï½C½b> loss/SRqqt3/assignment_loss"trainer/global_step2:JP)ŠC|@¾b; loss/FBt4/detection_loss"trainer/global_step2:JP%_ßÄ.¿) _wandbruntime‚50  _runtime‚50YSj8AÀ» /loss/FRt3/assignment_loss‚7.360267571040562 0loss/SRqqt2/assignment_loss‚5.45212869644165 .loss/FBt2/detection_loss‚6.349639985178198 -loss/FBt3/detection_loss‚5.47257917386014 /loss/FRt2/assignment_loss‚7.246345143569143 -loss/FRt2/detection_loss‚4.43043288863019 _runtime‚ 50.110243531 /loss/FBt1/assignment_loss‚2.723035156726837 /loss/FRt4/detection_loss‚3.5654784575193883 0loss/SRqqt3/detection_loss‚2.727877534155188 .loss/FBt1/detection_loss‚8.475902113318444 trainer/global_step‚49 % _timestamp‚1.7884165793603978e+09 /loss/FRt4/assignment_loss‚7.183738627332322 1loss/SRqqt4/assignment_loss‚5.402165742147536 /loss/FBt4/assignment_loss‚2.702881090102657 1loss/SRqqt4/detection_loss‚4.1610177636501335 epoch‚0 /loss/FRt1/assignment_loss‚7.279990341703771 /loss/FRt1/detection_loss‚2.8553006366147833 .loss/FRt3/detection_loss‚4.932797282934189 &loss/total_loss‚117.82546691484475 1loss/SRqqt1/assignment_loss‚5.315143315297253 0loss/SRqqt3/assignment_loss‚5.52151817288892 .loss/FBt4/detection_loss‚5.850313338541215 /loss/FBt2/assignment_loss‚2.731112939970834 0loss/SRqqt2/detection_loss‚4.202864074334502 /loss/FBt3/assignment_loss‚2.732225500047207 0loss/SRqqt1/detection_loss‚3.150711368839696 _step‚3tÞEÁ¿ /loss/FBt3/assignment_loss‚2.732225500047207 0loss/SRqqt1/detection_loss‚3.150711368839696 _step‚3 /loss/FRt3/assignment_loss‚7.360267571040562 0loss/SRqqt2/assignment_loss‚5.45212869644165 .loss/FBt2/detection_loss‚6.349639985178198 -loss/FBt3/detection_loss‚5.47257917386014 /loss/FRt2/assignment_loss‚7.246345143569143 -loss/FRt2/detection_loss‚4.43043288863019 _runtime‚ 50.110243531 /loss/FBt1/assignment_loss‚2.723035156726837 /loss/FRt4/detection_loss‚3.5654784575193883 0loss/SRqqt3/detection_loss‚2.727877534155188 .loss/FBt1/detection_loss‚8.475902113318444 trainer/global_step‚49 % _timestamp‚1.7884165793603978e+09 /loss/FRt4/assignment_loss‚7.183738627332322 1loss/SRqqt4/assignment_loss‚5.402165742147536 /loss/FBt4/assignment_loss‚2.702881090102657 1loss/SRqqt4/detection_loss‚4.1610177636501335 epoch‚0 /loss/FRt1/assignment_loss‚7.279990341703771 /loss/FRt1/detection_loss‚2.8553006366147833 .loss/FRt3/detection_loss‚4.932797282934189 &loss/total_loss‚117.82546691484475 1loss/SRqqt1/assignment_loss‚5.315143315297253 0loss/SRqqt3/assignment_loss‚5.52151817288892 .loss/FBt4/detection_loss‚5.850313338541215 /loss/FBt2/assignment_loss‚2.731112939970834 0loss/SRqqt2/detection_loss‚4.202864074334502™‹Â‚(:… ͤäÔ½œx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.662185668945312) gpu.0.memoryAllocatedBytes‚ 7440760832 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ygC´Ã‚(:ˆ ͤäÔ™þÕ{ disk.sda3.out‚ 2.05859375$ disk.nvme0c0n1.out‚ 2.86669921875 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚4 disk.dm-1.in‚0( proc.memory.availableMB‚ 354.39453125 disk.sda.out‚ 2.05859375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2755.53515625 proc.cpu.threads‚17 network.sent‚92558) disk./.usagePercent‚73.59385634552972 disk.rbd1.out‚ 0.06298828125 disk.dm-1.out‚ 2.05859375) proc.memory.percent‚16.81845188140869 disk.rbd3.in‚0 disk.nvme0n1.in‚4 disk.drbd1777.in‚0 disk.nvme0c0n1.in‚4 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 cpu‚24.988641445556922 disk.sda1.in‚0 disk.dm-0.out‚ 2.86669921875 network.recv‚113762 disk.rbd2.out‚4.875 disk.drbd1777.out‚0 disk.rbd4.in‚0% disk./.usageGB‚1096.6016426086426 disk.sda.in‚0" disk.nvme0n1.out‚ 2.86669921875 disk.rbd2.in‚0$ memory_percent‚97.836947441101075ÆúŽÄ‚(:ƒ ܤäÔ“ x gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.68' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JF Å‚(:ÿ ܤäÔÓÏÉ{ proc.cpu.threads‚17 disk.dm-0.in‚4 disk.sda1.in‚0 disk.nvme0n1.in‚4 disk.sda.in‚0 disk.sda3.out‚2.671875 network.recv‚120256) disk./.usagePercent‚73.58249268309987 disk.dm-0.out‚ 4.93603515625$ memory_percent‚97.83873558044434* proc.memory.percent‚16.818523406982422 network.sent‚142775$ disk./.usageGB‚1096.432315826416 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0! proc.memory.rssMB‚ 2755.546875 disk.rbd0.in‚0 disk.rbd1.out‚ 0.06298828125 disk.drbd1777.out‚0 disk.sda.out‚2.671875 disk.rbd3.in‚0 disk.dm-1.out‚2.671875 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4 disk.rbd1.in‚0" disk.nvme0n1.out‚ 4.93603515625 disk.rbd2.out‚4.875 disk.sda1.out‚0 disk.rbd3.out‚0' proc.memory.availableMB‚ 354.1015625 cpu‚24.99777065983515 disk.sda2.in‚0 disk.drbd1777.in‚0$ disk.nvme0c0n1.out‚ 4.93603515625:ûwÆ‚(:… ë¤äÔÉ´x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶¸×°Ç‚(:Œ ë¤äÔ¼âz disk.dm-1.out‚ 3.05859375 disk.rbd3.in‚0 disk.rbd1.out‚ 0.06298828125 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.nvme0c0n1.in‚4$ disk.nvme0c0n1.out‚ 5.39208984375 disk.sda.in‚0 disk.sda.out‚ 3.05859375* proc.memory.percent‚16.818571090698242) disk./.usagePercent‚73.58238490392796% disk./.usageGB‚1096.4307098388672 disk.sda2.in‚0 disk.rbd2.out‚ 6.5546875 cpu‚25.004852878051892 network.sent‚149757 network.recv‚125801 disk.rbd0.in‚0 disk.dm-0.in‚4 disk.rbd3.out‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2755.5546875 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 3.05859375" disk.nvme0n1.out‚ 5.39208984375# memory_percent‚97.8400468826294 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚4( proc.memory.availableMB‚ 353.88671875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.dm-0.out‚ 5.39208984375ª(Tm.È) _wandbruntime‚95  _runtime‚95mîĿɹ % lr-AdamW/pg1‚6.56985512280192e-05 % lr-AdamW/pg2‚6.56985512280192e-05 trainer/global_step‚99 % _timestamp‚1.7884166242685637e+09 _runtime‚ 95.018505386 _step‚4U×býÃʽ % lr-AdamW/pg1‚6.56985512280192e-05 % lr-AdamW/pg2‚6.56985512280192e-05 trainer/global_step‚99 % _timestamp‚1.7884166242685637e+09 _runtime‚ 95.018505386 _step‚4?VŸR.Ë) _wandbruntime‚95  _runtime‚95— LԨ̎O WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3850841459712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚647783 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4033533302784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4318044549120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 $ _timestamp‚1.788416624269289e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2574565 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 186849628160 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 182691843072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2550393 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3850859462656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1046772 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 186582136832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2550393 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 182919279616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4033778742272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 321957888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 182919279616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4033533302784 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 _step‚5 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1694555 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1230 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1784055 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1784055 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4358620 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4131486352384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4131462412288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1784053 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3850841459712 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 182691843072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291431424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚653 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚647132 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _runtime‚ 95.019385535 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4033778742272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3643466265600 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3825851826822 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚99 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2574565 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1693902 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3643449226240 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 182161919622 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0˜Ùâ˜'Í’O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1046770 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3825611145862 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4033778742272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3643466265600 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3825851826822 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚99 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2574565 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1693902 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3643449226240 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 182161919622 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3850841459712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚647783 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4033533302784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4318044549120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 $ _timestamp‚1.788416624269289e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2574565 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 186849628160 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 182691843072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2550393 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3850859462656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1046772 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 186582136832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2550393 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 182919279616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4033778742272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 321957888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 182919279616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4033533302784 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 _step‚5 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1694555 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1230 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1784055 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1784055 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4358620 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4131486352384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4131462412288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1784053 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3850841459712 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 182691843072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291431424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚653 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚647132 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _runtime‚ 95.019385535 HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 184529537536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3889197406720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3679751684096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚654386 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4073954394112 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3679768723456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4401916 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 184529537536 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3863746003318 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4377716 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1801772 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4361098055680 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1710840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1801774 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3889179403776 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4172656048640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4172632108544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2600142 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 321957888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4401916 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 184756987392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1801772 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 188737619456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2600142 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 184756987392 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2575944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1057127 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3863986684486 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3889179403776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0œð>£&ÐM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1801772 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 188737619456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2600142 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 184756987392 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2575944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1057127 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3863986684486 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3889179403776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3889197406720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 183994319222 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4073708941312 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚653715 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 _runtime‚ 95.883963281 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 295612416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 _step‚6 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 184529537536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3889197406720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3679751684096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚654386 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4073954394112 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3679768723456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4401916 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 184529537536 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3863746003318 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4377716 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1801772 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4361098055680 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1710840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1801774 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3889179403776 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4172656048640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4172632108544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2600142 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 321957888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4401916 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 184756987392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0òèë.Ñ) _wandbruntime‚95  _runtime‚95ÌÂŒ­ÒÄ 0loss/SRqqt3/detection_loss‚1.901256613320272 % _timestamp‚1.7884166251350849e+09 1loss/SRqqt4/assignment_loss‚5.350627845188357 0loss/FBt3/assignment_loss‚2.4447066887565283 .loss/FRt3/detection_loss‚3.674050889585329 /loss/FRt2/assignment_loss‚7.308649977048238 /loss/FRt4/assignment_loss‚7.164459158392513 1loss/SRqqt1/assignment_loss‚5.383822959044884 1loss/SRqqt2/assignment_loss‚5.ksz447665020823479 trainer/global_step‚99 _step‚7 /loss/FBt4/detection_loss‚1.9857519554373408 &loss/total_loss‚119.54771073387644 1loss/SRqqt3/assignment_loss‚5.502977548326765 0loss/FBt4/assignment_loss‚2.3673575340755404 /loss/FRt1/detection_loss‚2.6437236142695926 .loss/FRt2/detection_loss‚4.876711360282368 1loss/SRqqt4/detection_loss‚3.1243041000276244 .loss/FBt2/detection_loss‚8.426017282361334 _runtime‚ 95.884898804 /loss/FRt3/assignment_loss‚7.404387919799142 /loss/FBt1/detection_loss‚11.996375992894173 .loss/FBt3/detection_loss‚7.959077251994091 epoch‚0 0loss/FRt1/assignment_loss‚7.3249857230264634 /loss/FBt1/assignment_loss‚2.363063999584743 /loss/FBt2/assignment_loss‚2.252928311174566 1loss/SRqqt1/detection_loss‚2.7626511364661415 .loss/FRt4/detection_loss‚4.401982224601451 0loss/SRqqt2/detection_loss‚5.480175627395511m ™NÓÈ epoch‚0 0loss/FRt1/assignment_loss‚7.3249857230264634 /loss/FBt1/assignment_loss‚2.363063999584743 /loss/FBt2/assignment_loss‚2.252928311174566 1loss/SRqqt1/detection_loss‚2.7626511364661415 .loss/FRt4/detection_loss‚4.401982224601451 0loss/SRqqt2/detection_loss‚5.480175627395511 0loss/SRqqt3/detection_loss‚1.901256613320272 % _timestamp‚1.7884166251350849e+09 1loss/SRqqt4/assignment_loss‚5.350627845188357 0loss/FBt3/assignment_loss‚2.4447066887565283 .loss/FRt3/detection_loss‚3.674050889585329 /loss/FRt2/assignment_loss‚7.308649977048238 /loss/FRt4/assignment_loss‚7.164459158392513 1loss/SRqqt1/assignment_loss‚5.383822959044884 1loss/SRqqt2/assignment_loss‚5.447665020823479 trainer/global_step‚99 _step‚7 /loss/FBt4/detection_loss‚1.9857519554373408 &loss/total_loss‚119.54771073387644 1loss/SRqqt3/assignment_loss‚5.502977548326765 0loss/FBt4/assignment_loss‚2.3673575340755404 /loss/FRt1/detection_loss‚2.6437236142695926 .loss/FRt2/detection_loss‚4.876711360282368 1loss/SRqqt4/detection_loss‚3.1243041000276244 .loss/FBt2/detection_loss‚8.426017282361334 _runtime‚ 95.884898804 /loss/FRt3/assignment_loss‚7.404387919799142 /loss/FBt1/detection_loss‚11.996375992894173 .loss/FBt3/detection_loss‚7.959077251994091×@rÔ‚(:… ú¤äÔ¤î¤x gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÀoÕ‚(:’ ú¤äÔ¯áäz network.sent‚156057 disk.sda3.out‚ 3.75390625$ memory_percent‚97.84307479858398% disk./.usageGB‚1096.5608024597168 disk.sda.out‚ 3.75390625 disk.rbd1.out‚ 0.06298828125 disk.dm-0.in‚4 disk.sda2.out‚0 disk.drbd1777.in‚0& proc.memory.availableMB‚ 353.390625 disk.rbd3.in‚0 disk.sda.in‚0! disk.dm-0.out‚139.57275390625 disk.dm-1.out‚ 3.75390625 disk.sda1.in‚0 cpu‚25.008097240423115 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2755.56640625 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚4 disk.drbd1777.out‚0* proc.memory.percent‚16.818642616271973 network.recv‚130233) disk./.usagePercent‚73.59111552886814$ disk.nvme0n1.out‚139.57275390625 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.rbd2.out‚ 6.5546875 disk.nvme0c0n1.in‚4& disk.nvme0c0n1.out‚139.57275390625Ð-³Ö‚(:… ‰¥äÔ£¶Çx gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.602' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a©´×‚(:Š ‰¥äÔÞáÝ{) disk./.usagePercent‚73.59984768985355# disk./.usageGB‚1096.69091796875 disk.dm-0.in‚4 disk.dm-0.out‚273.0830078125 disk.sda2.in‚0 disk.rbd4.out‚0 disk.sda1.in‚0$ proc.memory.availableMB‚353.0625 disk.sda.out‚ 4.25390625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 network.sent‚206279 disk.rbd1.in‚0 disk.rbd1.out‚ 0.06298828125# disk.nvme0n1.out‚273.0830078125 disk.rbd0.out‚0 disk.rbd4.in‚0* proc.memory.percent‚16.818666458129883 cpu‚25.012051271075272 disk.dm-1.out‚ 4.25390625 disk.rbd2.in‚0" proc.memory.rssMB‚ 2755.5703125 network.recv‚136564 disk.rbd2.out‚ 8.1796875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚4 disk.sda3.out‚ 4.25390625 disk.nvme0n1.in‚4 disk.sda2.out‚0$ memory_percent‚97.84507751464844 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd3.out‚0% disk.nvme0c0n1.out‚273.0830078125yB‘IØ‚(:„ ˜¥äÔë–x gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.111' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%Ø£qüÙ‚(:ñ ˜¥äÔ®ö±{ disk.dm-1.out‚4.6875 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚4 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 0.06298828125 disk.nvme0n1.in‚4* proc.memory.percent‚16.818666458129883 network.sent‚211993 network.recv‚140285 disk.rbd2.out‚ 9.69140625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-0.out‚273.9375 disk.sda2.out‚0$ disk./.usageGB‚1096.690517425537 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.59982080906245 disk.sda.in‚0 disk.sda.out‚4.6875 disk.sda3.in‚0 disk.dm-0.in‚4( proc.memory.availableMB‚ 353.27734375 disk.nvme0c0n1.out‚273.9375 disk.rbd3.out‚0 cpu‚25.01414162404586 disk.rbd4.in‚0 disk.sda3.out‚4.6875 disk.nvme0n1.out‚273.9375$ memory_percent‚97.84376621246338" proc.memory.rssMB‚ 2755.5703125ò*Ù†0Ú+ _wandbruntime‚142  _runtime‚142¤ ƸÃÛ½ & lr-AdamW/pg1‚9.887963770681677e-05 & lr-AdamW/pg2‚9.887963770681677e-05 trainer/global_step‚149 % _timestamp‚1.7884166714201615e+09 _runtime‚ 142.170150979 _step‚8.½ÇÜÁ & lr-AdamW/pg1‚9.887963770681677e-05 & lr-AdamW/pg2‚9.887963770681677e-05 trainer/global_step‚149 % _timestamp‚1.7884166714201615e+09 _runtime‚ 142.170150979 _step‚8ˆÇC0Ý+ _wandbruntime‚142  _runtime‚142Ž¿VX—'Þ‘O P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6042329643008 trainer/global_step‚149 _runtime‚ 142.171196309 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2537799 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1564522 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 274590980096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5767738662912 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6525472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6189947225088 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5732367609870 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 284543488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6501298 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5767756665856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6470118573568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚972612 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 280171348480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5732608290830 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5458572119040 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1564520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3831295 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5767756665856 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 274019132430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3831295 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2537132 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2670003 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚973277 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6042575082496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2670005 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 274818416640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5458589158400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5767738662912 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 274590980096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289334272 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 280436742656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 273795490830 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2670005 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6189971165184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 _step‚9 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265394176 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2670003 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 6042575082496 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1213Õ+Æ>›'ß•O XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6470118573568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚972612 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 280171348480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5732608290830 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5458572119040 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1564520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3831295 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5767756665856 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 274019132430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3831295 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2537132 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2670003 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚973277 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6042575082496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2670005 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 274818416640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5458589158400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5767738662912 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 274590980096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289334272 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 280436742656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 273795490830 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2670005 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6189971165184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 _step‚9 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265394176 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2670003 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 6042575082496 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1213 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6042329643008 trainer/global_step‚149 _runtime‚ 142.171196309 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2537799 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1564522 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 274590980096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5767738662912 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6525472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6189947225088 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5732367609870 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 284543488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6501298 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5767756665856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 c Šj0à+ _wandbruntime‚143  _runtime‚143,M\|á›M M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5806094609920 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1574877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 _runtime‚ 143.056435552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2687722 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5806076606976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2553837 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3856846 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1574875 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2687724 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289320960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5770743148494 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 276656124416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚616 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1231 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6544568 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 276428674560 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3881044 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚978962 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6568768 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5494874576896 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 276656124416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5806094609920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 trainer/global_step‚149 _step‚10 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6544568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3881044 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 6082750734336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2554453 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5770502467326 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1213 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 6082750734336 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6568768 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5494874576896 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 276656124416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5806094609920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 trainer/global_step‚149 _step‚10 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6544568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3881044 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 6082750734336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2554453 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5770502467326 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1213 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 6082750734336 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5806094609920 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1574877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 _runtime‚ 143.056435552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2687722 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5806076606976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2553837 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3856846 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1574875 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2687724 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289320960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5770743148494 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 276656124416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚616 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1231 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6544568 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 276428674560 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3881044 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚978962 Ì=Ñ0ã+ _wandbruntime‚143  _runtime‚143“°vMäÇ _runtime‚ 143.057761501 epoch‚0 /loss/FRt2/assignment_loss‚7.331237771775988 0loss/FRt3/assignment_loss‚7.3252990113364325 1loss/SRqqt1/assignment_loss‚4.722840453639175 0loss/FBt1/assignment_loss‚0.9521189434453845 /loss/FRt2/detection_loss‚3.7417957021130457 .loss/FRt4/detection_loss‚6.022712246938185 1loss/SRqqt2/assignment_loss‚4.742087936401367 0loss/FBt4/assignment_loss‚0.6491488268742195 .loss/FBt1/detection_loss‚10.30942676961422 .loss/FBt3/detection_loss‚12.94989608653954 /loss/FBt4/detection_loss‚1.1281284208671982 0loss/FRt1/assignment_loss‚7.1556504890322685 1loss/SRqqt3/assignment_loss‚5.076227042985999 0loss/FBt2/assignment_loss‚0.4170750588751756 0loss/SRqqt2/detection_loss‚3.685615396499634 .loss/FBt2/detection_loss‚6.961771037716132 % _timestamp‚1.7884166723079083e+09 1loss/SRqqt4/assignment_loss‚4.817914401783662 0loss/FBt3/assignment_loss‚1.1400949997561318 .loss/FRt1/detection_loss‚2.440916120482143 0loss/SRqqt4/detection_loss‚4.923110423280912 &loss/total_loss‚113.31612704206425 _step‚11 0loss/SRqqt3/detection_loss‚2.448083537230559 .loss/FRt4/assignment_loss‚7.12101114879955 .loss/FRt3/detection_loss‚4.764858044270012 1loss/SRqqt1/detection_loss‚2.4891071718073254 trainer/global_step‚149®ÑQåË 1loss/SRqqt3/assignment_loss‚5.076227042985999 0loss/FBt2/assignment_loss‚0.4170750588751756 0loss/SRqqt2/detection_loss‚3.685615396499634 .loss/FBt2/detection_loss‚6.961771037716132 % _timestamp‚1.7884166723079083e+09 1loss/SRqqt4/assignment_loss‚4.817914401783662 0loss/FBt3/assignment_loss‚1.1400949997561318 .loss/FRt1/detection_loss‚2.440916120482143 0loss/SRqqt4/detection_loss‚4.923110423280912 &loss/total_loss‚113.31612704206425 _step‚11 0loss/SRqqt3/detection_loss‚2.448083537230559 .loss/FRt4/assignment_loss‚7.12101114879955 .loss/FRt3/detection_loss‚4.764858044270012 1loss/SRqqt1/detection_loss‚2.4891071718073254 trainer/global_step‚149 _runtime‚ 143.057761501 epoch‚0 /loss/FRt2/assignment_loss‚7.331237771775988 0loss/FRt3/assignment_loss‚7.3252990113364325 1loss/SRqqt1/assignment_loss‚4.722840453639175 0loss/FBt1/assignment_loss‚0.9521189434453845 /loss/FRt2/detection_loss‚3.7417957021130457 .loss/FRt4/detection_loss‚6.022712246938185 1loss/SRqqt2/assignment_loss‚4.742087936401367 0loss/FBt4/assignment_loss‚0.6491488268742195 .loss/FBt1/detection_loss‚10.30942676961422 .loss/FBt3/detection_loss‚12.94989608653954 /loss/FBt4/detection_loss‚1.1281284208671982 0loss/FRt1/assignment_loss‚7.1556504890322685 É`˜æ‚(:‚ §¥äÔÏ€ºx gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚96.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h~Ö¼ç‚(:‹ §¥äÔÙ…{ disk.nvme0n1.in‚4 disk.rbd2.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.818666458129883 disk.nvme0c0n1.in‚4 disk.sda.in‚0 disk.sda3.out‚ 5.22265625 cpu‚25.015882546144635 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚4 disk.dm-0.out‚ 274.20703125 disk.dm-1.in‚0# disk.nvme0c0n1.out‚ 274.20703125 disk.sda1.in‚0) disk./.usagePercent‚73.59983284141657 disk.rbd1.out‚ 0.06298828125 disk.rbd2.out‚ 9.85546875 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 5.22265625 disk.sda2.out‚0 network.sent‚218487 network.recv‚145398% disk./.usageGB‚1096.6906967163086 disk.drbd1777.in‚0! disk.nvme0n1.out‚ 274.20703125 disk.dm-1.out‚ 5.22265625 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd1.in‚0$ memory_percent‚97.84414768218994( proc.memory.availableMB‚ 353.21484375" proc.memory.rssMB‚ 2755.5703125 proc.cpu.threads‚17 disk.sda3.in‚0*+.öè‚(:… ¶¥äÔ¾¤x gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-ËZÛé‚(:‰ ¶¥äÔšëÕz disk.rbd4.in‚0* proc.memory.percent‚16.818666458129883) disk./.usagePercent‚73.59982285712273 disk.rbd3.out‚0 disk.sda2.in‚0 cpu‚25.018648042140104 disk.sda3.in‚0 disk.rbd1.in‚0! disk.dm-0.out‚274.74755859375 disk.dm-1.in‚0 disk.rbd2.out‚ 11.3671875 disk.drbd1777.in‚0 disk.rbd4.out‚0& disk.nvme0c0n1.out‚274.74755859375% disk./.usageGB‚1096.6905479431152 disk.nvme0n1.in‚4$ disk.nvme0n1.out‚274.74755859375$ memory_percent‚97.84440994262695& proc.memory.availableMB‚ 353.171875" proc.memory.rssMB‚ 2755.5703125 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda.out‚5.71875 disk.rbd2.in‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-0.in‚4 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚150945 disk.sda3.out‚5.71875 disk.nvme0c0n1.in‚4 network.sent‚268286 disk.sda1.out‚0 disk.rbd1.out‚ 0.06298828125 disk.dm-1.out‚5.71875¸Pê‚(:… Å¥äÔ™¯¥x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó²ë‚(: Å¥äÔÅôÂ{ network.recv‚156000 proc.cpu.threads‚17 disk.sda3.out‚ 6.59765625! disk.dm-0.out‚283.21630859375 network.sent‚275114 disk.sda1.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 6.59765625 disk.drbd1777.in‚0 cpu‚25.0207126292802( disk./.usagePercent‚73.6003684091782% disk./.usageGB‚1096.6986770629883 disk.rbd4.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 352.6171875" proc.memory.rssMB‚ 2755.5703125* proc.memory.percent‚16.818666458129883& disk.nvme0c0n1.out‚283.21630859375 disk.rbd3.in‚0 disk.rbd1.out‚ 0.06298828125$ disk.nvme0n1.out‚283.21630859375 disk.rbd0.out‚0# memory_percent‚97.8477954864502 disk.nvme0c0n1.in‚4 disk.rbd3.out‚0 disk.sda.out‚ 6.59765625 disk.sda3.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.sda.in‚0 disk.dm-0.in‚4 disk.rbd2.in‚0 disk.nvme0n1.in‚4 disk.sda2.out‚0 disk.rbd2.out‚ 11.4765625 disk.drbd1777.out‚0_l“«0ì+ _wandbruntime‚188  _runtime‚188ÆYÆíÀ trainer/global_step‚199 % _timestamp‚1.7884167173478396e+09 _runtime‚ 188.097794674 _step‚12 ' lr-AdamW/pg1‚0.00013206072418561434 ' lr-AdamW/pg2‚0.00013206072418561434ÿ3¸ÊîÄ ' lr-AdamW/pg1‚0.00013206072418561434 ' lr-AdamW/pg2‚0.00013206072418561434 trainer/global_step‚199 % _timestamp‚1.7884167173478396e+09 _runtime‚ 188.097794674 _step‚12 ðq0ï+ _wandbruntime‚188  _runtime‚188}tš'ð”O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 366717553664 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8051371422720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 374312774656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2082272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 % _timestamp‚1.7884167173485947e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 _runtime‚ 188.098723625 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8051125983232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 284543488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7684635866112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7684635866112 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 366490117120 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7273712051200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1233 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8248432037888 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 366717553664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3379681 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1297409 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2082270 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7273695011840 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 365652703638 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 365429062038 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3379031 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5136369 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3555955 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7639124073878 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3555953 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8622473126912 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8248455977984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚13 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7684653869056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271685632 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8051125983232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5112197 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5112197 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271685632 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8051125983232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5112197 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5112197 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2082272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 % _timestamp‚1.7884167173485947e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 _runtime‚ 188.098723625 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8051125983232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 284543488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7684635866112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7684635866112 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 366490117120 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7273712051200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1233 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8248432037888 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 366717553664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3379681 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1297409 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2082270 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7273695011840 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 365652703638 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 365429062038 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3379031 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5136369 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ÿ¯ûÛ0ò+ _wandbruntime‚188  _runtime‚188P¨µt¤&óžM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5161946 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8091547074560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7722973810176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1215 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1303093 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2092625 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7722991813120 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 368555261440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8735620 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5137748 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8665519275008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7722991813120 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 368327811584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2092627 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 _runtime‚ 188.961462912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5161946 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5137748 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7722973810176 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8711420 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3573674 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3395718 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 280932352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 367485103446 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8289625674240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1303719 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7677499612502 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8735620 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3573672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 368327811584 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 376174533120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2©FAT¨&ô¢M SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5161946 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5137748 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7722973810176 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8711420 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3573674 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3395718 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 280932352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 367485103446 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8289625674240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1303719 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7677499612502 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8735620 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3573672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 368327811584 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 376174533120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5161946 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8091547074560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7722973810176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1215 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1303093 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2092625 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7722991813120 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 368555261440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8735620 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5137748 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8665519275008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7722991813120 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 368327811584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2092627 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 _runtime‚ 188.961462912…™å0õ+ _wandbruntime‚188  _runtime‚188`ö”‚LöÆ 0loss/FBt4/assignment_loss‚0.6523094139178284 /loss/FRt4/detection_loss‚3.9950568887495224 0loss/SRqqt3/detection_loss‚2.667592930017209 0loss/FRt3/assignment_loss‚7.4138328366809425 2loss/SRqqt3/assignment_loss‚3.9251882885441636 2loss/SRqqt4/assignment_loss‚3.6181144131554497 /loss/SRqqt2/detection_loss‚2.22735720172976 trainer/global_step‚199 0loss/FRt1/assignment_loss‚7.2586425214573955 .loss/FRt3/detection_loss‚4.756245580812295 .loss/FBt1/detection_loss‚8.135048499703407 /loss/FRt2/assignment_loss‚7.386171731081876 0loss/FBt1/assignment_loss‚0.5592313312226906 0loss/FBt3/assignment_loss‚0.7345356866858448 %loss/total_loss‚92.85909639344254 _step‚15 .loss/FRt1/detection_loss‚2.410903081729792 -loss/FRt2/detection_loss‚2.92262106009505 /loss/FRt4/assignment_loss‚7.251739471189437 0loss/SRqqt4/detection_loss‚3.461960695683956 .loss/FBt3/detection_loss‚5.119820214582212 _runtime‚ 188.962457827 2loss/SRqqt1/assignment_loss‚3.4537490528899353 0loss/FBt2/assignment_loss‚0.4460489542596042 1loss/SRqqt1/detection_loss‚1.5869491827891922 .loss/FBt4/detection_loss‚2.042463820770334 epoch‚0 # _timestamp‚1.78841671821266e+09 2loss/SRqqt2/assignment_loss‚3.8458308414979414 .loss/FBt2/detection_loss‚6.987682694196701 š¢)P÷Ê 0loss/FRt3/assignment_loss‚7.4138328366809425 2loss/SRqqt3/assignment_loss‚3.9251882885441636 2loss/SRqqt4/assignment_loss‚3.6181144131554497 /loss/SRqqt2/detection_loss‚2.22735720172976 trainer/global_step‚199 0loss/FRt1/assignment_loss‚7.2586425214573955 .loss/FRt3/detection_loss‚4.756245580812295 .loss/FBt1/detection_loss‚8.135048499703407 /loss/FRt2/assignment_loss‚7.386171731081876 0loss/FBt1/assignment_loss‚0.5592313312226906 0loss/FBt3/assignment_loss‚0.7345356866858448 %loss/total_loss‚92.85909639344254 _step‚15 .loss/FRt1/detection_loss‚2.410903081729792 -loss/FRt2/detection_loss‚2.92262106009505 /loss/FRt4/assignment_loss‚7.251739471189437 0loss/SRqqt4/detection_loss‚3.461960695683956 .loss/FBt3/detection_loss‚5.119820214582212 _runtime‚ 188.962457827 2loss/SRqqt1/assignment_loss‚3.4537490528899353 0loss/FBt2/assignment_loss‚0.4460489542596042 1loss/SRqqt1/detection_loss‚1.5869491827891922 .loss/FBt4/detection_loss‚2.042463820770334 epoch‚0 # _timestamp‚1.78841671821266e+09 2loss/SRqqt2/assignment_loss‚3.8458308414979414 .loss/FBt2/detection_loss‚6.987682694196701 0loss/FBt4/assignment_loss‚0.6523094139178284 /loss/FRt4/detection_loss‚3.9950568887495224 0loss/SRqqt3/detection_loss‚2.667592930017209ªðøXø‚(:„ Ô¥äÔ†õÕx gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚97.578' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.5156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctϽp§7edMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A)Ä ù‚(:ø Ô¥äÔ‰Ž„| disk.rbd4.out‚0 disk.dm-0.out‚284.3876953125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 cpu‚25.02257586428737) disk./.usagePercent‚73.60037762544943 disk.drbd1777.out‚0% disk.nvme0c0n1.out‚284.3876953125 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚7.25" proc.memory.availableMB‚354.75 disk.rbd3.in‚0 disk.nvme0n1.in‚4 disk.sda1.out‚0 disk.sda3.in‚0# disk.nvme0n1.out‚284.3876953125 disk.dm-0.in‚4 disk.rbd0.out‚0$ memory_percent‚97.83477783203125! proc.memory.rssMB‚ 2755.578125 proc.cpu.threads‚17 network.sent‚280861 network.recv‚159978% disk./.usageGB‚1096.6988143920898 disk.sda.in‚0 disk.sda3.out‚7.25 disk.rbd2.in‚0 disk.rbd2.out‚ 12.98828125 disk.sda.out‚7.25 disk.drbd1777.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚4 disk.rbd1.out‚ 0.06298828125 disk.sda2.out‚0* proc.memory.percent‚16.818714141845703 B›ú‚(: ã¥äÔ¥ô¬x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E·Mû‚(: ã¥äÔ•óÍz disk.rbd1.out‚ 0.06298828125# disk.nvme0n1.out‚284.6337890625 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 7.70703125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0$ memory_percent‚97.83563613891602 disk.nvme0c0n1.in‚4 disk.sda3.out‚ 7.70703125 disk.dm-0.in‚4 disk.dm-0.out‚284.6337890625 disk.dm-1.in‚0# proc.memory.rssMB‚ 2755.58203125) disk./.usagePercent‚73.60038146556244 disk.rbd3.in‚0 disk.nvme0n1.in‚4* proc.memory.percent‚16.818737983703613 network.sent‚330978% disk.nvme0c0n1.out‚284.6337890625 disk.sda1.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 network.recv‚166252 disk.rbd2.out‚ 13.15234375 disk.rbd0.in‚0 disk.rbd4.out‚0& proc.memory.availableMB‚ 354.609375% disk./.usageGB‚1096.6988716125488 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 cpu‚25.02357059217399 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 7.70703125¼¹Áü‚(:„ ò¥äÔ÷Ž®x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.186' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£Ìý‚(:‹ ò¥äÔ¼é•{( proc.memory.availableMB‚ 354.82421875) disk./.usagePercent‚73.60906421709373 disk.dm-0.in‚4 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚171189 disk.sda3.out‚ 8.29296875 disk.dm-0.out‚ 389.19921875 disk.rbd2.out‚ 14.6640625 disk.drbd1777.in‚0 cpu‚25.02602534327644 disk.nvme0n1.in‚4 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 network.sent‚337326 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.06298828125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4% disk./.usageGB‚1096.8282508850098 disk.sda.in‚0 disk.sda.out‚ 8.29296875 disk.rbd4.out‚0* proc.memory.percent‚16.818737983703613 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd3.in‚0! disk.nvme0n1.out‚ 389.19921875 disk.sda2.out‚0# disk.nvme0c0n1.out‚ 389.19921875# proc.memory.rssMB‚ 2755.58203125 disk.sda1.out‚0 disk.dm-1.out‚ 8.29296875$ memory_percent‚97.83432483673096-ï¿Í0þ+ _wandbruntime‚233  _runtime‚233áÞÆÿÀ % _timestamp‚1.7884167631449878e+09 _runtime‚ 233.895008586 _step‚16 ' lr-AdamW/pg1‚0.00016524181066441193 ' lr-AdamW/pg2‚0.00016524181066441193 trainer/global_step‚249!ÍÕïÊ€Ä _step‚16 ' lr-AdamW/pg1‚0.00016524181066441193 ' lr-AdamW/pg2‚0.00016524181066441193 trainer/global_step‚249 % _timestamp‚1.7884167631449878e+09 _runtime‚ 233.895008586Jºm0+ _wandbruntime‚233  _runtime‚233oº×ߦ'‚ O XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _step‚17 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10059922323456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9601551072256 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1621636 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10306916850688 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9545880537886 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10059922323456 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4441903 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2600022 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 457286274846 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4221656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 % _timestamp‚1.7884167631457272e+09 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6417271 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4222283 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10859176 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4441905 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1233 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 9601533069312 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 458616690688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _runtime‚ 233.895880719 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 458389254144 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10775009526272 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6417271 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9546121218846 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 458616690688 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚10060167762944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6393099°Ü¡ª'ƒ¤O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9601533069312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 468068735488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269588480 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10835002 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10059922323456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9601551072256 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1621636 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10306916850688 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9545880537886 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10059922323456 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4441903 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2600022 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 457286274846 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4221656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 % _timestamp‚1.7884167631457272e+09 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6417271 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4222283 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10859176 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4441905 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1233 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 9601533069312 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 458616690688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _runtime‚ 233.895880719 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 458389254144 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10775009526272 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6417271 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9546121218846 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 458616690688 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚10060167762944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6393099 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _step‚17 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚614 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10348110487040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9125137401856 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4459622 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4238762 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9584256076510 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6418650 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2610377 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2610375 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 460226948608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 _step‚18 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4459624 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4459622 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10348086546944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 460454398464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 458895032846 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10902472 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4238146 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6442848 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9584015395342 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9639889016320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 469664319488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4459624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9639871013376 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6442848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 459118674654 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10878272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6418650 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9639889016320 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚10100343414784 _runtime‚ 234.753256417 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 458895032846 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10902472 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4238146 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6442848 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9584015395342 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9639889016320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 469664319488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4459624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9639871013376 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6442848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 459118674654 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10878272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6418650 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9639889016320 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚10100343414784 _runtime‚ 234.753256417 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚614 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10348110487040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9125137401856 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4459622 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4238762 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9584256076510 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6418650 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2610377 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2610375 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 460226948608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 _step‚18 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4459624 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4459622 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10348086546944òi‡0‡+ _wandbruntime‚234  _runtime‚234%\0OˆÉ /loss/FRt1/detection_loss‚2.5918470146292347 _runtime‚ 234.754401244 0loss/FBt1/assignment_loss‚0.3378701051438434 -loss/FBt1/detection_loss‚5.84035717640762 /loss/FBt2/detection_loss‚7.3269081957962205 &loss/total_loss‚102.88072059119585 epoch‚0 .loss/FRt2/detection_loss‚4.042600701749325 1loss/SRqqt1/assignment_loss‚3.224648960849695 0loss/SRqqt3/assignment_loss‚3.55766590152468 .loss/FRt3/detection_loss‚4.054819206520915 1loss/SRqqt4/detection_loss‚3.5999210539569213 /loss/FBt4/detection_loss‚0.8621383859972798 /loss/FRt4/detection_loss‚3.7947572355086985 _step‚19 /loss/FRt2/assignment_loss‚7.330537569522858 0loss/SRqqt3/detection_loss‚4.689918089375454 .loss/FRt4/assignment_loss‚7.08265156012315 0loss/FBt2/assignment_loss‚0.4463168920584671 0loss/FBt3/assignment_loss‚0.6668423187220469 % _timestamp‚1.7884167640045872e+09 /loss/FRt3/assignment_loss‚7.338389253616333 2loss/SRqqt2/assignment_loss‚3.3681506792704266 0loss/FBt4/assignment_loss‚0.4464004929995665 /loss/FRt1/assignment_loss‚7.125891152074781 1loss/SRqqt1/detection_loss‚2.3218684543144743 1loss/SRqqt2/detection_loss‚3.3814509727888638 /loss/FBt3/detection_loss‚15.915692519057881 trainer/global_step‚249 1loss/SRqqt4/assignment_loss‚3.533076699187116aÌsLS‰Í 1loss/SRqqt4/assignment_loss‚3.533076699187116 /loss/FRt1/detection_loss‚2.5918470146292347 _runtime‚ 234.754401244 0loss/FBt1/assignment_loss‚0.3378701051438434 -loss/FBt1/detection_loss‚5.84035717640762 /loss/FBt2/detection_loss‚7.3269081957962205 &loss/total_loss‚102.88072059119585 epoch‚0 .loss/FRt2/detection_loss‚4.042600701749325 1loss/SRqqt1/assignment_loss‚3.224648960849695 0loss/SRqqt3/assignment_loss‚3.55766590152468 .loss/FRt3/detection_loss‚4.054819206520915 1loss/SRqqt4/detection_loss‚3.5999210539569213 /loss/FBt4/detection_loss‚0.8621383859972798 /loss/FRt4/detection_loss‚3.7947572355086985 _step‚19 /loss/FRt2/assignment_loss‚7.330537569522858 0loss/SRqqt3/detection_loss‚4.689918089375454 .loss/FRt4/assignment_loss‚7.08265156012315 0loss/FBt2/assignment_loss‚0.4463168920584671 0loss/FBt3/assignment_loss‚0.6668423187220469 % _timestamp‚1.7884167640045872e+09 /loss/FRt3/assignment_loss‚7.338389253616333 2loss/SRqqt2/assignment_loss‚3.3681506792704266 0loss/FBt4/assignment_loss‚0.4464004929995665 /loss/FRt1/assignment_loss‚7.125891152074781 1loss/SRqqt1/detection_loss‚2.3218684543144743 1loss/SRqqt2/detection_loss‚3.3814509727888638 /loss/FBt3/detection_loss‚15.915692519057881 trainer/global_step‚249ÛTõŽŠ‚(:ƒ ¦äÔ“Ä•x gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.72' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.544 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šò‹‚(:ù ¦äÔý½Ûz% disk./.usageGB‚1096.9533081054688 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda2.out‚0) disk./.usagePercent‚73.61745691208873 disk.nvme0c0n1.out‚ 488.34375 disk.rbd3.out‚0 disk.sda3.out‚8.84375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 network.sent‚344046 disk.rbd1.in‚0* proc.memory.percent‚16.818737983703613 disk.sda1.out‚0 disk.dm-1.out‚8.84375 disk.sda.out‚8.84375 disk.nvme0n1.out‚ 488.34375 disk.dm-0.in‚4 disk.dm-0.out‚ 488.34375$ memory_percent‚97.83616065979004 cpu‚25.02652586338601 network.recv‚176621 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.nvme0c0n1.in‚4 disk.sda.in‚0 disk.rbd1.out‚ 0.06298828125 disk.nvme0n1.in‚4 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2755.58203125 proc.cpu.threads‚17 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd2.out‚ 16.34765625 disk.rbd0.out‚0' proc.memory.availableMB‚ 354.5234375 disk.dm-1.in‚0×Ú™Œ‚(:Ž ¦äÔóÉx gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.648' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.729599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç}‚(: ¦äÔµå¯z disk.rbd1.out‚ 0.06298828125 disk.rbd2.out‚ 16.34765625 disk.sda3.in‚0 network.recv‚182084 disk.nvme0n1.in‚4 disk.drbd1777.out‚0 disk.sda1.in‚0$ memory_percent‚97.83315658569336& proc.memory.availableMB‚ 355.015625 proc.cpu.threads‚17 disk.dm-1.in‚0& disk.nvme0c0n1.out‚489.05615234375 disk.sda.in‚0 disk.sda.out‚ 9.4296875! disk.dm-0.out‚489.05615234375 disk.sda2.in‚0 network.sent‚393547) disk./.usagePercent‚73.61745332798326 disk.dm-0.in‚4 disk.rbd4.in‚0* proc.memory.percent‚16.818737983703613 disk.dm-1.out‚ 9.4296875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd3.out‚0 cpu‚25.02695164377758 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 9.4296875# proc.memory.rssMB‚ 2755.58203125 disk.sda2.out‚0$ disk./.usageGB‚1096.953254699707$ disk.nvme0n1.out‚489.05615234375 disk.drbd1777.in‚0 disk.nvme0c0n1.in‚4]•3Z›Ž‚(: Ÿ¦äÔˆ÷x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚110.267' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.053399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=\ ‚(:• Ÿ¦äÔ³ƒÞ{ disk.dm-0.in‚4 disk.rbd2.in‚0 network.recv‚187146 disk.rbd3.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚489.30615234375 disk.rbd2.out‚ 17.9609375 proc.cpu.threads‚17% disk./.usageGB‚1096.9533348083496 disk.sda.in‚0 disk.nvme0n1.in‚4 disk.dm-1.out‚ 9.86328125 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 9.86328125 disk.rbd0.in‚0 cpu‚25.028723076864388 network.sent‚399848 disk.sda2.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2755.58984375) disk./.usagePercent‚73.61745870414147& disk.nvme0c0n1.out‚489.30615234375 disk.sda3.in‚0! disk.dm-0.out‚489.30615234375 disk.sda2.in‚0* proc.memory.percent‚16.818785667419434 disk.rbd4.in‚0 disk.rbd1.out‚ 0.06298828125 disk.dm-1.in‚0$ memory_percent‚97.83689975738525( proc.memory.availableMB‚ 354.40234375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4 disk.sda3.out‚ 9.86328125B4§’0+ _wandbruntime‚279  _runtime‚279œÔüÄ‘¾ _step‚20 & lr-AdamW/pg1‚0.0001984228971432095 & lr-AdamW/pg2‚0.0001984228971432095 trainer/global_step‚299 % _timestamp‚1.7884168088291347e+09 _runtime‚ 279.579054135^ÃÄðÈ’Â & lr-AdamW/pg2‚0.0001984228971432095 trainer/global_step‚299 % _timestamp‚1.7884168088291347e+09 _runtime‚ 279.579054135 _step‚20 & lr-AdamW/pg1‚0.0001984228971432095í),)0“+ _wandbruntime‚279  _runtime‚279L5ë®'”¨O ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5327853 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12068718663680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7698173 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3117772 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 548696204454 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 250714112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 % _timestamp‚1.7884168088298354e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1215 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 561741477888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13001854 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12365401663488 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12068964103168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 548919846054 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11452637001894 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12365425603584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3117770 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11518430272512 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13026028 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5062703 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12068718663680 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7698173 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 _runtime‚ 279.580014104 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5327855 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11518430272512 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 550288391168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12068964103168 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12927167081472 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11518448275456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 284543488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1944933 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10903957836800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 550515827712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5327855 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5327853 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 550515827712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1945564 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11452877682854 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 _step‚21 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11518448275456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7674001 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5063336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 274654208 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12926892427264 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1233øÉ4yR •¬O L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12365425603584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3117770 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11518430272512 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13026028 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5062703 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12068718663680 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7698173 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 _runtime‚ 279.580014104 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5327855 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11518430272512 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 550288391168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12068964103168 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12927167081472 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11518448275456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 284543488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batc/&÷`h_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1944933 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10903957836800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 550515827712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5327855 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5327853 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 550515827712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1945564 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11452877682854 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 _step‚21 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11518448275456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7674001 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5063336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 274654208 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12926892427264 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1233 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5327853 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12068718663680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7698173 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3117772 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 548696204454 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 250714112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 % _timestamp‚1.7884168088298354e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1215 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 561741477888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13001854 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12365401663488 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12068964103168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 548919846054 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11452637001894 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262Ê@Xb0–+ _wandbruntime‚280  _runtime‚280Ž5¸&—²M MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5345574 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 552353535488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7699552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 552126085632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1952019 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12109139755008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13045124 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 550752245862 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12108894302208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚610 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12406595299840 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1215 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7723750 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 563632385536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚22 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5079534 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 563371198976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285126656 _runtime‚ 280.454059495 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5345572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261186560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚10940243255296 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 550528604054 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12109139755008 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7723750 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13069324 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 % _timestamp‚1.7884168097029095e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 552126085632 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7723750 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13069324 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 % _timestamp‚1.7884168097029095e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 552126085632 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7699552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 552126085632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1952019 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12109139755008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13045124 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 550752245862 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12108894302208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚610 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12406595299840 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1215 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7723750 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 563632385536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚22 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5079534 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 563371198976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285126656 _runtime‚ 280.454059495 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5345572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261186560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824!lZ0™+ _wandbruntime‚280  _runtime‚280Áö¿DMšÇ %loss/total_loss‚92.59291378255747 _step‚23 2loss/SRqqt4/assignment_loss‚3.0150296568870543 1loss/FBt1/assignment_loss‚0.17109618372180396 1loss/SRqqt1/detection_loss‚2.1160126591718647 /loss/FBt1/detection_loss‚3.7739908597949476 /loss/FBt4/detection_loss‚2.9418174296915724 1loss/FBt3/assignment_loss‚0.35592824672090095 0loss/FBt4/assignment_loss‚0.5954447269726258 /loss/FRt2/detection_loss‚3.5420889383138614 1loss/SRqqt2/detection_loss‚2.7493375890087663 0loss/SRqqt3/detection_loss‚2.463631303679056 _runtime‚ 280.455066663 /loss/FRt2/assignment_loss‚7.291705053906108 1loss/SRqqt2/assignment_loss‚3.235660377301668 /loss/FBt2/assignment_loss‚0.490510493342299 -loss/FRt1/detection_loss‚2.42042896603093 epoch‚0 trainer/global_step‚299 0loss/SRqqt1/assignment_loss‚3.42923382543168 /loss/FRt3/detection_loss‚3.5135323184987772 1loss/SRqqt4/detection_loss‚3.1410868222955606 /loss/FRt3/assignment_loss‚7.402025720347529 .loss/FBt3/detection_loss‚9.579349098934067 $ _timestamp‚1.788416809705214e+09 /loss/FRt4/assignment_loss‚7.269017267227173 2loss/SRqqt3/assignment_loss‚3.1397109354956676 .loss/FRt4/detection_loss‚5.132312256097793 ,loss/FBt2/detection_loss‚7.8081205870424 /loss/FRt1/assignment_loss‚7.015842466643362ÍfŒQ›Ë /loss/FRt3/assignment_loss‚7.402025720347529 .loss/FBt3/detection_loss‚9.579349098934067 $ _timestamp‚1.788416809705214e+09 /loss/FRt4/assignment_loss‚7.269017267227173 2loss/SRqqt3/assignment_loss‚3.1397109354956676 .loss/FRt4/detection_loss‚5.132312256097793 ,loss/FBt2/detection_loss‚7.8081205870424 /loss/FRt1/assignment_loss‚7.015842466643362 %loss/total_loss‚92.59291378255747 _step‚23 2loss/SRqqt4/assignment_loss‚3.0150296568870543 1loss/FBt1/assignment_loss‚0.17109618372180396 1loss/SRqqt1/detection_loss‚2.1160126591718647 /loss/FBt1/detection_loss‚3.7739908597949476 /loss/FBt4/detection_loss‚2.9418174296915724 1loss/FBt3/assignment_loss‚0.35592824672090095 0loss/FBt4/assignment_loss‚0.5954447269726258 /loss/FRt2/detection_loss‚3.5420889383138614 1loss/SRqqt2/detection_loss‚2.7493375890087663 0loss/SRqqt3/detection_loss‚2.463631303679056 _runtime‚ 280.455066663 /loss/FRt2/assignment_loss‚7.291705053906108 1loss/SRqqt2/assignment_loss‚3.235660377301668 /loss/FBt2/assignment_loss‚0.490510493342299 -loss/FRt1/detection_loss‚2.42042896603093 epoch‚0 trainer/global_step‚299 0loss/SRqqt1/assignment_loss‚3.42923382543168 /loss/FRt3/detection_loss‚3.5135323184987772 1loss/SRqqt4/detection_loss‚3.1410868222955606ošR˜›œ‚(: ®¦äÔ‡÷¤x gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.651' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.330199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à[‚(:‘ ®¦äÔ¶¨Ÿ{) disk./.usagePercent‚73.63423436584517 disk.sda3.out‚ 10.5078125 disk.rbd1.in‚0$ disk.nvme0n1.out‚589.23095703125 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 10.5078125 disk.rbd4.out‚0 cpu‚25.029025780741968 disk.rbd1.out‚ 0.06298828125 disk.dm-0.in‚4 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2755.58984375 network.recv‚191482% disk./.usageGB‚1097.2033042907715$ memory_percent‚97.83773422241211& proc.memory.availableMB‚ 354.265625 network.sent‚406026 disk.sda.in‚0! disk.dm-0.out‚589.23095703125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 10.5078125 disk.rbd2.out‚17.96875 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚4& disk.nvme0c0n1.out‚589.23095703125 disk.nvme0n1.in‚4* proc.memory.percent‚16.818785667419434~ëñ'ž‚(:„ ½¦äÔÎéÕx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.562' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=JI›Ÿ‚(: ½¦äÔܘ¶{ disk.sda3.in‚0 disk.rbd2.in‚0 cpu‚25.02996075267348 network.sent‚456206 disk.nvme0c0n1.in‚4 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.06298828125 disk.nvme0n1.in‚4 disk.sda2.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 354.22265625 disk.rbd4.in‚0& disk.nvme0c0n1.out‚589.49267578125 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd2.out‚ 19.6015625$ memory_percent‚97.83799648284912 disk.rbd0.out‚0! disk.dm-0.out‚589.49267578125* proc.memory.percent‚16.818809509277344$ disk./.usageGB‚1097.203384399414 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0$ disk.nvme0n1.out‚589.49267578125 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 10.984375 proc.cpu.threads‚17 disk.sda3.out‚ 10.984375 disk.dm-0.in‚4 disk.dm-1.out‚ 10.9843 âm}75 proc.memory.rssMB‚ 2755.59375 network.recv‚197623) disk./.usagePercent‚73.63423974200339 disk.rbd4.out‚0•ô ‚(:… ̦äÔƒï™x gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.872' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç-€H¡‚(:ˆ ̦äÔæçÀz network.sent‚461734 disk.sda.out‚ 11.734375 disk.dm-0.in‚4 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4 disk.nvme0n1.in‚4 disk.rbd1.in‚0 disk.rbd1.out‚ 0.06298828125 disk.dm-1.in‚0 disk.rbd2.out‚ 19.6015625 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 network.recv‚201131 disk.rbd2.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.818857192993164 cpu‚25.031712978739378 disk.rbd4.in‚0) disk./.usagePercent‚73.63425100633489 disk.rbd4.out‚0$ disk.nvme0c0n1.out‚ 688.705078125 disk.rbd3.in‚0" disk.nvme0n1.out‚ 688.705078125% disk./.usageGB‚1097.2035522460938 disk.drbd1777.out‚0 disk.sda3.out‚ 11.734375$ memory_percent‚97.83687591552734 disk.dm-1.out‚ 11.734375 disk.drbd1777.in‚0 disk.sda.in‚0 disk.dm-0.out‚ 688.705078125% proc.memory.availableMB‚ 354.40625" proc.memory.rssMB‚ 2755.6015625H¡0¢+ _wandbruntime‚325  _runtime‚325:IèFÅ£¿ ' lr-AdamW/pg2‚0.00023160398362200707 trainer/global_step‚349 % _timestamp‚1.7884168548424537e+09 _runtime‚ 325.59238933 _step‚24 ' lr-AdamW/pg1‚0.00023160398362200707ñÁɤà trainer/global_step‚349 % _timestamp‚1.7884168548424537e+09 _runtime‚ 325.59238933 _step‚24 ' lr-AdamW/pg1‚0.00023160398362200707 ' lr-AdamW/pg2‚0.00023160398362200707r~¿Ç0¥+ _wandbruntime‚325  _runtime‚325®6®'¦¨O ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 640553417262 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6213803 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 642414964736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 284543488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14077760443392 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15079266566144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚12719080729600 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15168706 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14077515003904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15079557997568 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8979075 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14077760443392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13435327475712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13435345478656 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15192880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8954903 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 642187528192 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 642187528192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2270731 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12719063690240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚647 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚14423886476288 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14077515003904 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 655380089856 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6213805 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5906253 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14423910416384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3635520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 % _timestamp‚1.7884168548432565e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2270084 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1215 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 325.593527154 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291431424 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8954903 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5905604 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13435345478656 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6213803 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 640329775662 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6213805 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚349 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13359393465902 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 642414964736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13435327475712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14077515003904 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 655380089856 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6213805 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5906253 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14423910416384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3635520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 % _timestamp‚1.7884168548432565e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2270084 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1215 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 325.593527154 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291431424 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8954903 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5905604 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13435345478656 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6213803 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 640329775662 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6213805 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚349 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13359393465902 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 642414964736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13435327475712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14077760443392 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15079266566144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0{Þ&0¨+ _wandbruntime‚326  _runtime‚326Z£;¸&©²M MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6231524 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 644252672512 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14465080112640 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12755366148096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15236176 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14117936095232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚12755383187456 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5922949 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3645877 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13397769004526 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13473665419776 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6231524 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9004652 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15122597553152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9004652 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14117936095232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 293515264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1215 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13473665419776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8980454 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14117690642432 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5922309 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6231522 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15236176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15211976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14465056172544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15211976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 644025222656 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 269575168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6231522 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1233 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 644025222656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _step‚26 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884168559608881e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13473683422720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 657517440512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8980454 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13473683422720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15122304037888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12755366148096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15236176 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14117936095232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚12755383187456 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5922949 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3645877 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13397769004526 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13473665419776 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6231524 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9004652 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15122597553152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9004652 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14117936095232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 293515264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1215 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13473665419776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8980454 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14117690642432 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5922309 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6231522 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15236176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15211976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14465056172544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15211976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 644025222656 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 269575168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6231522 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1233 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 644025222656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _step‚26 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884168559608881e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13473683422720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 657517440512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0Ôí¬0«+ _wandbruntime‚326  _runtime‚326ÀAü{J¬Ä .loss/FRt2/detection_loss‚3.490710973739624 /loss/FRt1/assignment_loss‚5.481967436300741 1loss/SRqqt1/detection_loss‚3.9464559209009407 _step‚27 1loss/SRqqt1/assignment_loss‚3.406012459805137 .loss/FRt4/detection_loss‚4.162324734032154 1loss/SRqqt2/detection_loss‚1.8881259571760893 /loss/FBt2/detection_loss‚11.224177877817835 .loss/FBt3/detection_loss‚4.454385634173047 _runtime‚ 326.712754265 /loss/FRt2/assignment_loss‚4.273735258890235 /loss/FRt4/assignment_loss‚6.996843636035919 1loss/SRqqt4/detection_loss‚3.5677225139405992 %loss/total_loss‚90.92014559371381 % _timestamp‚1.7884168559628294e+09 2loss/SRqqt3/assignment_loss‚3.3227238410558457 trainer/global_step‚349 /loss/FRt3/assignment_loss‚7.063125168404928 0loss/SRqqt2/assignment_loss‚2.72698860168457 0loss/FBt1/assignment_loss‚0.4561585868090136 /loss/FRt1/detection_loss‚2.2670586654866063 epoch‚0 0loss/SRqqt4/assignment_loss‚2.37329539987776 0loss/FBt4/assignment_loss‚0.5356475276386609 /loss/FBt2/assignment_loss‚0.518750358302246 .loss/FRt3/detection_loss‚3.907965335177212 .loss/FBt1/detection_loss‚3.218705289243607 .loss/FBt4/detection_loss‚6.959007379963346 0loss/FBt3/assignment_loss‚0.4232328512042648 /loss/SRqqt3/detection_loss‚4.255024186053427ížrN­È /loss/FRt2/assignment_loss‚4.273735258890235 /loss/FRt4/assignment_loss‚6.996843636035919 1loss/SRqqt4/detection_loss‚3.5677225139405992 %loss/total_loss‚90.92014559371381 % _timestamp‚1.7884168559628294e+09 2loss/SRqqt3/assignment_loss‚3.3227238410558457 trainer/global_step‚349 /loss/FRt3/assignment_loss‚7.063125168404928 0loss/SRqqt2/assignment_loss‚2.72698860168457 0loss/FBt1/assignment_loss‚0.4561585868090136 /loss/FRt1/detection_loss‚2.2670586654866063 epoch‚0 0loss/SRqqt4/assignment_loss‚2.37329539987776 0loss/FBt4/assignment_loss‚0.5356475276386609 /loss/FBt2/assignment_loss‚0.518750358302246 .loss/FRt3/detection_loss‚3.907965335177212 .loss/FBt1/detection_loss‚3.218705289243607 .loss/FBt4/detection_loss‚6.959007379963346 0loss/FBt3/assignment_loss‚0.4232328512042648 /loss/SRqqt3/detection_loss‚4.25502418605342 .loss/FRt2/detection_loss‚3.490710973739624 /loss/FRt1/assignment_loss‚5.481967436300741 1loss/SRqqt1/detection_loss‚3.9464559209009407 _step‚27 1loss/SRqqt1/assignment_loss‚3.406012459805137 .loss/FRt4/detection_loss‚4.162324734032154 1loss/SRqqt2/detection_loss‚1.8881259571760893 /loss/FBt2/detection_loss‚11.224177877817835 .loss/FBt3/detection_loss‚4.454385634173047 _runtime‚ 326.712754265>]y®‚(:… Û¦äÔ«ã¢x gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.073' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.6146 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùØS÷¯‚(:’ Û¦äÔîÖÆz disk.rbd1.out‚ 0.06298828125 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.sda3.out‚ 12.0703125 disk.rbd0.in‚0 disk.rbd4.in‚0% proc.memory.availableMB‚ 354.34375 proc.cpu.threads‚17 network.sent‚468099% disk./.usageGB‚1097.2035789489746 disk.dm-1.in‚0# memory_percent‚97.8372573852539* proc.memory.percent‚16.818881034851074 disk.nvme0n1.in‚4 disk.dm-1.out‚ 12.0703125 disk.rbd3.out‚0) disk./.usagePercent‚73.63425279838764$ disk.nvme0n1.out‚787.84716796875 disk.sda2.out‚0 disk.rbd2.in‚0 cpu‚25.032475875669224! disk.dm-0.out‚787.84716796875 disk.rbd4.out‚0& disk.nvme0c0n1.out‚787.84716796875 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda.out‚ 12.0703125# proc.memory.rssMB‚ 2755.60546875 network.recv‚206126 disk.rbd2.out‚ 21.29296875 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚4 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚4 disk.rbd0.out‚0×ç‚š°‚(: ê¦äÔé«“x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.46' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.491999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»‹÷ ±‚(:• ê¦äÔÆø˜{ network.sent‚517812 disk.rbd2.out‚ 21.29296875 disk.rbd4.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚887.5048828125 disk.sda3.in‚0 disk.nvme0n1.in‚4# proc.memory.rssMB‚ 2755.60546875 disk.dm-1.out‚ 12.71484375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 0.06298828125# disk.nvme0n1.out‚887.5048828125 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.nvme0c0n1.in‚4* proc.memory.percent‚16.818881034851074% disk./.usageGB‚1097.7037467956543 disk.rbd4.out‚0$ memory_percent‚97.83883094787598 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 12.71484375' proc.memory.availableMB‚ 354.0859375 cpu‚25.033094681396342 proc.cpu.threads‚17) disk./.usagePercent‚73.66781948224708 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda.out‚ 12.71484375 disk.rbd1.in‚0 disk.dm-0.out‚887.5048828125 network.recv‚211417 disk.dm-0.in‚4ò:=²‚(:„ ù¦äÔ¯ œx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.731' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.1462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºmçH³‚(:” ù¦äÔ½îï{) disk./.usagePercent‚73.66782153030736 disk.sda2.out‚0 disk.rbd2.out‚ 22.97265625 disk.nvme0c0n1.in‚4 disk.rbd3.out‚0 proc.cpu.threads‚17 network.recv‚215743% disk./.usageGB‚1097.7037773132324 disk.sda1.in‚0 disk.sda.out‚ 13.19140625 disk.sda.in‚0 network.sent‚524245 disk.rbd1.in‚0 disk.dm-0.in‚4 cpu‚25.034305949270664 disk.sda1.out‚0 disk.nvme0n1.in‚4 disk.dm-1.out‚ 13.19140625 disk.rbd0.out‚0 disk.rbd4.in‚0$ memory_percent‚97.84250259399414 disk.sda3.out‚ 13.19140625 disk.rbd1.out‚ 0.06298828125 disk.dm-0.out‚887.7431640625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0% disk.nvme0c0n1.out‚887.7431640625 disk.drbd1777.in‚0 disk.rbd3.in‚0& proc.memory.availableMB‚ 353.484375# proc.memory.rssMB‚ 2755.60546875* proc.memory.percent‚16.818881034851074 disk.sda3.in‚0# disk.nvme0n1.out‚887.7431640625 disk.dm-1.in‚0 disk.rbd2.in‚0°æD 0´+ _wandbruntime‚372  _runtime‚372‹'‹wŵ¿ ' lr-AdamW/pg2‚0.00026478507010080463 trainer/global_step‚399 $ _timestamp‚1.788416901327675e+09 _runtime‚ 372.077579967 _step‚28 ' lr-AdamW/pg1‚0.00026478507010080463„•)“ɶà ' lr-AdamW/pg2‚0.00026478507010080463 trainer/global_step‚399 $ _timestamp‚1.788416901327675e+09 _runtime‚ 372.077579967 _step‚28 ' lr-AdamW/pg1‚0.00026478507010080463û϶0·+ _wandbruntime‚372  _runtime‚372Ï_»"²'¸¬O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6747616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289334272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 732186988470 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15352224678912 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 749452988416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265394176 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 731963346870 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 734314101760 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7099755 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17231848217600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15352242681856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17359732 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10259977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15266390610870 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16482371289088 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10235805 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 trainer/global_step‚399 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16086311344128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2594344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16086556783616 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 734086665216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16482395229184 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14534186583040n·­¹°O O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17231558883328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4153272 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6746977 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7099753 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16086556783616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17335558 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10235805 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 trainer/global_step‚399 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16086311344128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2594344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16086556783616 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 734086665216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16482395229184 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14534186583040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6747616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289334272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 732186988470 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15352224678912 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 749452988416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265394176 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 731963346870 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 734314101760 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7099755 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17231848217600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15352242681856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17359732 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10259977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15266390610870 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_s&Ò plit_bytes.large_pool.freed‚16482371289088 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17403028 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17274603266048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 trainer/global_step‚399 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10285554 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16126732435456 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4163625 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 751327661568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14570489040896 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2601254 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 751062280704 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 733795746470 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15390562622976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2600594 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265380864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15304284787366 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 734019388278 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16523564925440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 372.928155985 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17378828 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10285554 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15304525468534 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚660 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7117474 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7117472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15390580625920 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6764219 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4163627 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 736151809536 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16126486982656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10261356 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17274892587008 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 735924359680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7117474 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10261356 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚15390580625920 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 735924359680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14570506080256 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17378828 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15390562622976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17403028 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1250 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16523540985344 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16126732435456 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _step‚30 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7117472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚662 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7117474 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10261356 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚15390580625920 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 735924359680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14570506080256 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17378828 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15390562622976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17403028 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1250 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16523540985344 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16126732435456 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _step‚30 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7117472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚662 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17403028 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17274603266048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 trainer/global_step‚399 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10285554 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16126732435456 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4163625 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 751327661568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14570489040896 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2601254 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 751062280704 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 733795746470 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15390562622976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2600594 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265380864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15304284787366 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 734019388278 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16523564925440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 372.928155985 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17378828 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10285554 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15304525468534 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚660 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7117474 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7117472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15390580625920 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6764219 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4163627 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 736151809536 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16126486982656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10261356 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17274892587008 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299º™0½+ _wandbruntime‚372  _runtime‚372ЃîØM¾Ç 0loss/FRt1/assignment_loss‚5.7129820585250854 0loss/FBt2/assignment_loss‚0.3220630383236442 0loss/FBt4/assignment_loss‚0.3795415861044175 .loss/FRt2/detection_loss‚2.714829823688457 0loss/SRqqt2/detection_loss‚1.590876130641184 %loss/total_loss‚85.33416807364446 1loss/FBt3/assignment_loss‚0.20379553449038212 .loss/FBt3/detection_loss‚4.893074099905789 _runtime‚ 372.929153364 /loss/FRt3/assignment_loss‚6.644015678992639 /loss/FRt1/detection_loss‚2.9394795830760683 0loss/SRqqt4/detection_loss‚3.927502739887971 -loss/FBt1/detection_loss‚2.37640805973608 % _timestamp‚1.7884169021793547e+09 .loss/FBt4/detection_loss‚8.019468347815906 /loss/FRt2/assignment_loss‚4.718769148776405 /loss/FRt4/assignment_loss‚6.667561504575941 2loss/SRqqt2/assignment_loss‚2.4846157634800132 0loss/SRqqt3/detection_loss‚4.534489864652807 .loss/FRt3/detection_loss‚3.941165335667439 .loss/FBt2/detection_loss‚5.101412960461208 epoch‚0 _step‚31 2loss/SRqqt3/assignment_loss‚2.8173390879775537 0loss/FBt1/assignment_loss‚0.5124856580739781 0loss/SRqqt1/detection_loss‚5.189863203501399 trainer/global_step‚399 2loss/SRqqt1/assignment_loss‚3.4031495314378004 1loss/SRqqt4/assignment_loss‚2.597696732252072 .loss/FRt4/detection_loss‚3.641582601600223Îj}Q¿Ë 2loss/SRqqt1/assignment_loss‚3.4031495314378004 1loss/SRqqt4/assignment_loss‚2.597696732252072 .loss/FRt4/detection_loss‚3.641582601600223 0loss/FRt1/assignment_loss‚5.7129820585250854 0loss/FBt2/assignment_loss‚0.3220630383236442 0loss/FBt4/assignment_loss‚0.3795415861044175 .loss/FRt2/detection_loss‚2.714829823688457 0loss/SRqqt2/detection_loss‚1.590876130641184 %loss/total_loss‚85.33416807364446 1loss/FBt3/assignment_loss‚0.20379553449038212 .loss/FBt3/detection_loss‚4.893074099905789 _runtime‚ 372.929153364 /loss/FRt3/assignment_loss‚6.644015678992639 /loss/FRt1/detection_loss‚2.9394795830760683 0loss/SRqqt4/detection_loss‚3.927502739887971 -loss/FBt1/detection_loss‚2.37640805973608 % _timestamp‚1.7884169021793547e+09 .loss/FBt4/detection_loss‚8.019468347815906 /loss/FRt2/assignment_loss‚4.718769148776405 /loss/FRt4/assignment_loss‚6.667561504575941 2loss/SRqqt2/assignment_loss‚2.4846157634800132 0loss/SRqqt3/detection_loss‚4.534489864652807 .loss/FRt3/detection_loss‚3.941165335667439 .loss/FBt2/detection_loss‚5.101412960461208 epoch‚0 _step‚31 2loss/SRqqt3/assignment_loss‚2.8173390879775537 0loss/FBt1/assignment_loss‚0.5124856580739781 0loss/SRqqt1/detection_loss‚5.189863203501399 trainer/global_step‚399,†³é›À‚(: ˆ§äÔÁì¢x gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.356' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.871199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R¡|RÁ‚(:“ ˆ§äÔÆî®| proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.sda.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0& disk.nvme0c0n1.out‚987.33837890625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 13.96484375 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 13.96484375 network.sent‚529238) disk./.usagePercent‚73.66783279463887 disk.nvme0c0n1.in‚4 disk.nvme0n1.in‚4 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda3.in‚0! disk.dm-0.out‚987.33837890625# memory_percent‚97.8398323059082 cpu‚25.034828105699575 network.recv‚219615 disk.rbd2.out‚ 24.484375 disk.rbd3.in‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 353.921875* proc.memory.percent‚16.818881034851074 disk.rbd1.in‚0 disk.rbd1.out‚ 0.06298828125 disk.dm-0.in‚4# proc.memory.rssMB‚ 2755.60546875$ disk./.usageGB‚1097.703945159912 disk.dm-1.in‚0 disk.dm-1.out‚ 13.96484375 disk.rbd3.out‚0$ disk.nvme0n1.out‚987.33837890625hšV¾Â‚(:„ —§äÔïí˜x gpu.0.gpu‚20.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.392' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t hâÂ(: —§äÔÙðÔ{ disk.sda3.out‚ 14.3515625 disk.sda2.in‚0 disk.nvme0c0n1.in‚4# proc.memory.rssMB‚ 2755.68359375 cpu‚25.03486311901513 proc.cpu.threads‚17( disk./.usagePercent‚73.6678353547142% disk./.usageGB‚1097.7039833068848 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.rbd4.out‚0' proc.memory.availableMB‚ 353.8359375 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0* proc.memory.percent‚16.819357872009277! disk.dm-0.out‚1086.4384765625 disk.dm-1.out‚ 14.3515625 disk.rbd0.out‚0 disk.sda.in‚0 disk.rbd2.out‚24.65625 disk.sda3.in‚0$ memory_percent‚97.84035682678223& disk.nvme0c0n1.out‚1086.4384765625 network.sent‚579346 disk.rbd1.out‚ 0.06298828125$ disk.nvme0n1.out‚1086.4384765625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 network.recv‚225593 disk.nvme0n1.in‚4 disk.dm-0.in‚4 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 14.3515625àeÄ‚(:… ¦§äÔ̼x gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.266' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}†€¸ Å‚(:€ ¦§äÔÂȵ{ disk.sda2.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4 disk.rbd3.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2755.68359375 disk.nvme0n1.in‚4 disk.rbd3.in‚0 disk.sda.out‚14.9375 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚14.9375 disk.rbd4.in‚0# disk.nvme0c0n1.out‚ 1185.9140625% disk./.usageGB‚1097.7039260864258 disk.sda3.out‚14.9375$ memory_percent‚97.84040451049805 network.recv‚229938 disk.sda2.in‚0 disk.rbd2.out‚ 26.16796875 disk.sda1.in‚0 disk.rbd1.out‚ 0.06298828125* proc.memory.percent‚16.819357872009277 proc.cpu.threads‚17 network.sent‚585527 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda1.out‚0! disk.nvme0n1.out‚ 1185.9140625 disk.dm-0.in‚4 cpu‚25.0354176300043 disk.sda.in‚0 disk.dm-0.out‚ 1185.9140625& proc.memory.availableMB‚ 353.828125 disk.drbd1777.out‚0) disk./.usagePercent‚73.66783151460119ª|È0Æ+ _wandbruntime‚417  _runtime‚417kâ`²ÆÇÀ trainer/global_step‚449 % _timestamp‚1.7884169472068312e+09 _runtime‚ 417.956754356 _step‚32 ' lr-AdamW/pg1‚0.00029796615657960224 ' lr-AdamW/pg2‚0.00029796615657960224ãê³mÊÈÄ _step‚32 ' lr-AdamW/pg1‚0.00029796615657960224 ' lr-AdamW/pg2‚0.00029796615657960224 trainer/global_step‚449 % _timestamp‚1.7884169472068312e+09 _runtime‚ 417.956754356 zKð0É+ _wandbruntime‚417  _runtime‚4177æÿpʬO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _step‚33 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 KADeviceStatsMonitor.on_train_batch_s¨¹È¡&tart/active.small_pool.current‚24172 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚17269139885056 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 trainer/global_step‚449 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19383817855488 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2919316 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2919977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18095353123840 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4671022 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18095353123840 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 842961753600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 284543488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7590336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17269121882112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1250 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291431424 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 823820559678 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18095107684352 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 826213238784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16349326515200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4671020 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17269139885056 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19502410 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11516707 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19384109286912 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17173147074878 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18540880041984 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 825985802240 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7985703 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11516707 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 823596918078 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17269121882112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19526584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16349309475840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7985705 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 843229244928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267491328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18095107684352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884169472077265e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 843229244928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267491328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18095107684352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884169472077265e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18095107684352 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 826213238784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16349326515200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4671020 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17269139885056 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19502410 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11516707 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19384109286912 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17173147074878 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18540880041984 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 825985802240 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7985703 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11516707 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 823596918078 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17269121882112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19526584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16349309475840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7985705 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2!•¶‡40Ì+ _wandbruntime‚418  _runtime‚418"Õ:»&͵M D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚17307477829120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4681377 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2926249 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18135283322880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18135528775680 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚17211281932542 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 825652959486 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2925599 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18582025798144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11542258 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11542258 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17307477829120 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚34 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8003424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11566456 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 828050946560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚449 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7606974 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11566456 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18135528775680 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8003422 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16385628973056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19569880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 293515264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19545680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 827823496704 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16385611933696 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7607626 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17307459826176 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 845119556608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 844849981440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8003422 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17307459826176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19426875779584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17211041251374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19427169294848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 $ _timestamp‚1.788416948086808e+09M†Òê› Î¹M VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17307459826176 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 845119556608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 844849981440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8003422 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17307459826176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19426875779584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17211041251374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19427169294848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 $ _timestamp‚1.788416948086808e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ÃÞ¤4$17307477829120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4681377 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2926249 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18135283322880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18135528775680 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚17211281932542 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 825652959486 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2925599 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18582025798144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11542258 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11542258 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17307477829120 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚34 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8003424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11566456 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 828050946560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚449 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7606974 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11566456 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18135528775680 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8003422 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16385628973056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19569880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 293515264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19545680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 827823496704 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16385611933696 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7607626":« 0Ï+ _wandbruntime‚418  _runtime‚418Ò.cðQÐË epoch‚0 /loss/FRt2/assignment_loss‚4.401791848634419 1loss/SRqqt2/assignment_loss‚2.671828708383772 0loss/FBt1/assignment_loss‚0.6601765340892598 1loss/FBt4/assignment_loss‚0.41133254882273224 /loss/FRt3/detection_loss‚3.5721510900900912 trainer/global_step‚449 /loss/FRt3/assignment_loss‚5.540519469823593 /loss/FRt4/assignment_loss‚6.075285911560059 2loss/SRqqt3/assignment_loss‚1.9074378944173151 .loss/FBt3/detection_loss‚3.646146699786186 0loss/FBt2/assignment_loss‚0.3266564195558506 /loss/FRt1/detection_loss‚3.4772507930579395 0loss/SRqqt2/detection_loss‚1.561062055101825 1loss/SRqqt3/detection_loss‚2.8262053810212078 %loss/total_loss‚83.07573310308652 1loss/SRqqt1/assignment_loss‚3.384723703066508 .loss/FBt2/detection_loss‚4.731424374436891 /loss/FRt2/detection_loss‚2.6987733835713907 /loss/FRt1/assignment_loss‚6.045434381650842 0loss/SRqqt4/detection_loss‚2.971282965540886 _step‚35 1loss/FBt3/assignment_loss‚0.26178803189675537 $ _timestamp‚1.788416948088747e+09 2loss/SRqqt4/assignment_loss‚2.6991339468955995 .loss/FRt4/detection_loss‚3.279058074951172 _runtime‚ 418.838571438 1loss/SRqqt1/detection_loss‚3.6711553825686374 .loss/FBt1/detection_loss‚4.273513610328738 /loss/FBt4/detection_loss‚11.981599893834856µ i°UÑÏ 1loss/SRqqt1/detection_loss‚3.6711553825686374 .loss/FBt1/detection_loss‚4.273513610328738 /loss/FBt4/detection_loss‚11.981599893834856 epoch‚0 /loss/FRt2/assignment_loss‚4.401791848634419 1loss/SRqqt2/assignment_loss‚2.671828708383772 0loss/FBt1/assignment_loss‚0.6601765340892598 1loss/FBt4/assignment_loss‚0.41133254882273224 /loss/FRt3/detection_loss‚3.5721510900900912 trainer/global_step‚449 /loss/FRt3/assignment_loss‚5.540519469823593 /loss/FRt4/assignment_loss‚6.075285911560059 2loss/SRqqt3/assignment_loss‚1.9074378944173151 .loss/FBt3/detection_loss‚3.646146699786186 0loss/FBt2/assignment_loss‚0.3266564195558506 /loss/FRt1/detection_loss‚3.4772507930579395 0loss/SRqqt2/detection_loss‚1.561062055101825 1loss/SRqqt3/detection_loss‚2.8262053810212078 %loss/total_loss‚83.07573310308652 1loss/SRqqt1/assignment_loss‚3.384723703066508 .loss/FBt2/detection_loss‚4.731424374436891 /loss/FRt2/detection_loss‚2.6987733835713907 /loss/FRt1/assignment_loss‚6.045434381650842 0loss/SRqqt4/detection_loss‚2.971282965540886 _step‚35 1loss/FBt3/assignment_loss‚0.26178803189675537 $ _timestamp‚1.788416948088747e+09 2loss/SRqqt4/assignment_loss‚2.6991339468955995 .loss/FRt4/detection_loss‚3.279058074951172 _runtime‚ 418.838571438# ŒhÒ‚(:… µ§äÔ³‘°x gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥°(¶Ó‚(:‰ µ§äÔ쉞{ disk.rbd2.in‚0* proc.memory.percent‚16.819381713867188 proc.cpu.threads‚17 disk.sda.out‚ 15.39453125 disk.sda3.in‚0 disk.nvme0n1.in‚4 disk.rbd0.out‚0 cpu‚25.03592909959146 disk.nvme0c0n1.in‚4 disk.rbd1.in‚0 disk.rbd1.out‚ 0.06298828125! disk.nvme0n1.out‚ 1186.1484375 disk.dm-1.in‚0 disk.dm-1.out‚ 15.39453125 disk.rbd2.out‚ 26.27734375$ memory_percent‚97.84250259399414 proc.memory.rssMB‚ 2755.6875 network.recv‚235258 disk.rbd4.in‚0 disk.dm-0.out‚ 1186.1484375& proc.memory.availableMB‚ 353.484375 network.sent‚592129 disk.sda1.in‚0) disk./.usagePercent‚73.66784380296284 disk.drbd1777.in‚0 disk.drbd1777.out‚0# disk.nvme0c0n1.out‚ 1186.1484375 disk.rbd3.in‚0 disk.sda3.out‚ 15.39453125 disk.dm-0.in‚4 disk.sda2.out‚0% disk./.usageGB‚1097.7041091918945 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.sda2.in‚0w®äÔ‚(:… ħäÔͶx gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 'Õ‚(:“ ħäÔôêÂ{ disk.sda1.in‚0 disk.dm-0.in‚4 disk.drbd1777.in‚0# memory_percent‚97.8410005569458 disk.sda1.out‚0 disk.rbd3.out‚0$ disk.nvme0n1.out‚1285.4072265625 disk.dm-1.in‚0 disk.rbd4.in‚0( proc.memory.availableMB‚ 353.73046875 proc.memory.rssMB‚ 2755.6875 proc.cpu.threads‚17 network.recv‚240286% disk./.usageGB‚1097.6688041687012 disk.rbd4.out‚0) disk./.usagePercent‚73.66547445323344 disk.sda.out‚ 16.04296875 disk.sda3.out‚ 16.04296875* proc.memory.percent‚16.819381713867188 disk.dm-1.out‚ 16.04296875 disk.rbd0.out‚0 disk.rbd3.in‚0! disk.dm-0.out‚1285.4072265625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.06298828125 disk.drbd1777.out‚0& disk.nvme0c0n1.out‚1285.4072265625 disk.sda3.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 27.7890625 disk.sda.in‚0 disk.nvme0n1.in‚4 disk.rbd0.in‚0 cpu‚25.036354198145855 network.sent‚641258 disk.nvme0c0n1.in‚4õ2ó,Ö‚(:„ Ó§äÔöðÛx gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.471' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03•>-ׂ(: Ó§äÔñ²Àz cpu‚25.036894830216188 network.recv‚244903) disk./.usagePercent‚73.66548750961769 disk.rbd2.out‚ 27.89453125 disk.nvme0n1.in‚4$ memory_percent‚97.84250259399414 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd4.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚4 disk.rbd0.out‚0 disk.sda.out‚ 16.3984375$ disk.nvme0n1.out‚1384.6630859375* proc.memory.percent‚16.819381713867188 network.sent‚647507 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda3.in‚0& proc.memory.availableMB‚ 353.484375% disk./.usageGB‚1097.6689987182617 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4 disk.rbd3.in‚0 disk.sda3.out‚ 16.3984375 disk.rbd1.out‚ 0.06298828125 proc.memory.rssMB‚ 2755.6875! disk.dm-0.out‚1384.6630859375 disk.dm-1.out‚ 16.3984375 disk.drbd1777.out‚0 disk.rbd4.out‚0& disk.nvme0c0n1.out‚1384.6630859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0¥Fø0Ø+ _wandbruntime‚463  _runtime‚463%Z­[ÃÙ½ $ _timestamp‚1.788416993129412e+09 _runtime‚ 463.879372265 _step‚36 & lr-AdamW/pg1‚0.0003311472430583998 & lr-AdamW/pg2‚0.0003311472430583998 trainer/global_step‚499î;CsÇÚÁ _runtime‚ 463.879372265 _step‚36 & lr-AdamW/pg1‚0.0003311472430583998 & lr-AdamW/pg2‚0.0003311472430583998 trainer/global_step‚499 $ _timestamp‚1.788416993129412e+09$ [œC0Û+ _wandbruntime‚463  _runtime‚463Cüõ±'Ü«O H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 _step‚37 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3243952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265394176 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21693436 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 936765510144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20103904024576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 trainer/global_step‚499 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 $ _timestamp‚1.788416993130169e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19186019085312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19186037088256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 915454130886 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8871653 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 917884939264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12797609 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20103904024576 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 917884939264 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12821781 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8433407 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20599364854784 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20599340914688 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12797609 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21536106424832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19186019085312 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19079662857926 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19186019085312 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19079662857926 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 _step‚37 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3243952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265394176 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21693436 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 936765510144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚$P3¶ 20103904024576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 trainer/global_step‚499 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 $ _timestamp‚1.788416993130169e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19186019085312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19186037088256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 915454130886 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8871653 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 917884939264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12797609 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20103904024576 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 917884939264%Re*€0Þ+ _wandbruntime‚464  _runtime‚464¬ö×$¼&߶M QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 919950083584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 _step‚38 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 trainer/global_step‚499 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8889372 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8889374 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19224357029376 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 919950083584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5199127 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 917286530694 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 464.738479747 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12847358 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8889372 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20640534551040 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18200734826496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 % _timestamp‚1.7884169939877276e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12823160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21736732 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1232 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 938649877504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19224375032320 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 938923646976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273769472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21736732 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 919722633728 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19118038396550 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21712532 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3250272 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8889374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5199125 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3250916 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21579458198016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20640510610944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20144325115904 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21712532 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18200751865856 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21579160488448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12823160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19117797715382 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8449397?—+EÀ&àºM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20144325115904 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21712532 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18200751865856 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21579160488448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12823160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19117797715382 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8449397 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 919950083584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 _step‚38 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 trainer/global_step‚499 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8889372 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8889374 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19224357029376 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 919950083584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5199127 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 917286530694 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 464.738479747 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12847358 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8889372 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20640534551040 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18200734826496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 % _timestamp‚1.7884169939877276e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12823160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21736732 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1232 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 938649877504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19224375032320 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 938923646976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273769472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21736732 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 919722633728 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19118038396550 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21712532 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3250272 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8889374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5199125 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3250916 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21579458198016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20640510610944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824&½ÿpÛ0á+ _wandbruntime‚464  _runtime‚464À…VÏOâÉ /loss/FRt4/detection_loss‚3.0444143897011164 2loss/SRqqt1/assignment_loss‚3.0642568171024323 /loss/FRt2/detection_loss‚2.4168401767810184 $ _timestamp‚1.788416993989807e+09 .loss/FBt1/detection_loss‚1.173434841166278 trainer/global_step‚499 .loss/FRt1/assignment_loss‚6.07242779109789 /loss/FRt3/assignment_loss‚5.937809732225206 0loss/FBt1/assignment_loss‚0.8567635309696198 0loss/SRqqt3/detection_loss‚2.983107703072684 _runtime‚ 464.739642328 /loss/FRt4/assignment_loss‚6.376950513748896 1loss/SRqqt4/assignment_loss‚2.780705753299925 1loss/SRqqt1/detection_loss‚3.1932358369231224 1loss/SRqqt4/detection_loss‚3.7186496837271585 .loss/FBt2/detection_loss‚3.454747406875386 .loss/FBt3/detection_loss‚5.201999957540205 .loss/FRt2/assignment_loss‚3.95627939303716 2loss/SRqqt2/assignment_loss‚2.6918147908316716 0loss/FBt4/assignment_loss‚0.7380967736244202 /loss/FBt4/detection_loss‚12.127161651849747 epoch‚0 1loss/FBt3/assignment_loss‚0.19093971901241144 .loss/FRt1/detection_loss‚3.223440718391667 /loss/FRt3/detection_loss‚3.7116362518734403 1loss/SRqqt2/detection_loss‚1.1206350256068012 _step‚39 1loss/SRqqt3/assignment_loss‚1.983933789389474 %loss/total_loss‚80.39849435968245 0loss/FBt2/assignment_loss‚0.3792121118347174ë ËîSãÍ 2loss/SRqqt1/assignment_loss‚3.0642568171024323 /loss/FRt2/detection_loss‚2.4168401767810184 $ _timestamp‚1.788416993989807e+09 .loss/FBt1/detection_loss‚1.173434841166278 trainer/global_step‚499 .loss/FRt1/assignment_loss‚6.07242779109789 /loss/FRt3/assignment_loss‚5.937809732225206 0loss/FBt1/assignment_loss‚0.8567635309696198 0loss/SRqqt3/detection_loss‚2.983107703072684 _runtime‚ 464.739642328 /loss/FRt4/assignment_loss‚6.376950513748896 1loss/SRqqt4/assignment_loss‚2.780705753299925 1loss/SRqqt1/detection_loss‚3.1932358369231224 1loss/SRqqt4/detection_loss‚3.7186496837271585 .loss/FBt2/detection_loss‚3.454747406875386 .loss/FBt3/detection_loss‚5.201999957540205 .loss/FRt2/assignment_loss‚3.95627939303716 2loss/SRqqt2/assignment_loss‚2.6918147908316716 0loss/FBt4/assignment_loss‚0.7380967736244202 /loss/FBt4/detection_loss‚12.127161651849747 epoch‚0 1loss/FBt3/assignment_loss‚0.19093971901241144 .loss/FRt1/detection_loss‚3.223440718391667 /loss/FRt3/detection_loss‚3.7116362518734403 1loss/SRqqt2/detection_loss‚1.1206350256068012 _step‚39 1loss/SRqqt3/assignment_loss‚1.983933789389474 %loss/total_loss‚80.39849435968245 0loss/FBt2/assignment_loss‚0.3792121118347174 /loss/FRt4/detection_loss‚3.0444143897011164'é¹€ä‚(:… â§äÔÄû£x gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.688' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5376 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}09 å‚(:€ â§äÔö§€{ disk.drbd1777.in‚0 disk.rbd4.in‚0 proc.memory.rssMB‚ 2755.6875 proc.cpu.threads‚17 network.recv‚249529 disk.rbd0.out‚0! disk.nvme0c0n1.out‚ 1484.28125$ memory_percent‚97.84290790557861) disk./.usagePercent‚73.66548366950467 disk.sda2.in‚0 disk.rbd2.out‚29.40625 disk.rbd3.in‚0 disk.sda3.out‚ 16.9140625 disk.nvme0c0n1.in‚4 disk.sda1.out‚0 network.sent‚653782 disk.rbd0.in‚0 disk.nvme0n1.in‚4* proc.memory.percent‚16.819381713867188 disk.sda.out‚ 16.9140625 disk.rbd1.in‚0% disk./.usageGB‚1097.6689414978027 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda.in‚0 disk.dm-0.in‚4 disk.dm-1.out‚ 16.9140625 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 0.06298828125 disk.nvme0n1.out‚ 1484.28125 disk.dm-0.out‚ 1484.28125 disk.dm-1.in‚0( proc.memory.availableMB‚ 353.41796875 cpu‚25.036732882160152EgÍæ‚(:… ñ§äÔż•x gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚122.149' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôª/<ç‚(:‹ ñ§äÔ¿»üz& disk.nvme0c0n1.out‚1484.5986328125 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚73.66548750961769 disk.sda.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚17.46875 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.06298828125$ disk.nvme0n1.out‚1484.5986328125 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚4 disk.rbd3.in‚0 disk.sda.out‚17.46875' proc.memory.availableMB‚ 353.6484375 cpu‚25.037129529049736 network.recv‚255647 disk.rbd3.out‚0 disk.rbd2.out‚ 30.94140625$ memory_percent‚97.84150123596191 disk.sda3.out‚17.46875! disk.dm-0.out‚1484.5986328125 proc.memory.rssMB‚ 2755.6875 disk.sda2.out‚0 disk.dm-1.in‚0% disk./.usageGB‚1097.6689987182617 disk.dm-0.in‚4 network.sent‚703897 disk.rbd4.out‚0 disk.nvme0n1.in‚4* proc.memory.percent‚16.819381713867188­ü¥Öè‚(:‚ €¨äÔâ•Šx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.03 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fèé‚(: €¨äÔùà…{ disk.rbd3.out‚0 disk.sda3.out‚ 18.078125 disk.rbd1.in‚0 disk.nvme0n1.in‚4 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda.in‚0 disk.rbd2.in‚0* proc.memory.percent‚16.819381713867188 proc.cpu.threads‚17' disk.nvme0c0n1.out‚1584.39794921875 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd2.out‚ 31.08203125& proc.memory.availableMB‚ 353.640625 disk.rbd4.in‚0 disk.rbd1.out‚ 0.06298828125% disk./.usageGB‚1097.6686058044434 disk.drbd1777.in‚0 disk.dm-0.in‚4 disk.dm-1.out‚ 18.078125$ memory_percent‚97.84154891967773 cpu‚25.037971676378998" disk.dm-0.out‚1584.39794921875 proc.memory.rssMB‚ 2755.6875 disk.sda1.out‚0 disk.sda.out‚ 18.078125 network.sent‚709439 network.recv‚259259) disk./.usagePercent‚73.66546114084166 disk.drbd1777.out‚0 disk.sda3.in‚0% disk.nvme0n1.out‚1584.39794921875 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4ÀŠ*˜0ê+ _wandbruntime‚509  _runtime‚509Ž.þÚÆëÀ _step‚40 ' lr-AdamW/pg1‚0.00036432832953719736 ' lr-AdamW/pg2‚0.00036432832953719736 trainer/global_step‚549 % _timestamp‚1.7884170390389035e+09 _runtime‚ 509.788924588N üøÊìÄ ' lr-AdamW/pg1‚0.00036432832953719736 ' lr-AdamW/pg2‚0.00036432832953719736 trainer/global_step‚549 % _timestamp‚1.7884170390389035e+09 _runtime‚ 509.788924588 _step‚40(º¼4]0í+ _wandbruntime‚509  _runtime‚509PÄÊÊu î´O IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22112700364800 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22112945804288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22112700364800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚21102916288512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚296ˆí˜E70 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1250 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1030899433472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9757603 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9272368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22657825727488 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚20986660002894 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23688447183872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚ 509.789745074 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚21102934291456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1009784076288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1009784076288 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14102683 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23860288 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14102683 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20986419321934 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1007087702094 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9273052 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23836114 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14078511 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚19979572300800 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1010011512832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277977088 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 trainer/global_step‚549 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5706522 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19979555261440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3565848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22657849667584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5706520 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1030621456384 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22112945804288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301917184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1010011512832 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9757605 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚21102934291456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚21102916288512 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9757605 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23688749101056 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824‰VÈä¾'ï¸O XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23688447183872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚ 509.789745074 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚21102934291456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1009784076288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1009784076288 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14102683 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23860288 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14102683 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20986419321934 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1007087702094 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9273052 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23836114 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14078511 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚19979572300800 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1010011512832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277977088 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 trainer/global_step‚549 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5706522 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19979555261440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3565848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22657849667584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5706520 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1030621456384 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22112945804288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301917184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1010011512832 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9757605 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚21102934291456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚21102916288512 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9757605 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23688749101056 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22112700364800 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22112945804288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22112700364800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚21102916288512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1250 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1030899433472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9757603 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9272368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22657825727488 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚20986660002894 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0)?µ$šð‚(: ¨äÔ°šx gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.08' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.215999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“¯[ñ‚(: ¨äÔ˜ô¹{ disk.sda1.in‚0 disk.rbd3.out‚0$ disk.nvme0n1.out‚1683.5087890625 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.nvme0c0n1.in‚4& disk.nvme0c0n1.out‚1683.5087890625 disk.rbd3.in‚0 disk.sda.out‚ 18.546875 proc.cpu.threads‚17$ memory_percent‚97.84300327301025 proc.memory.rssMB‚ 2755.6875 disk.rbd4.out‚0 disk.rbd1.out‚ 0.06298828125 disk.dm-0.in‚4 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 18.546875 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 18.546875 network.recv‚263908% disk./.usageGB‚1097.6691398620605 disk.dm-1.in‚0 disk.rbd4.in‚0 disk.nvme0n1.in‚4 network.sent‚715688 disk.rbd2.out‚ 32.62109375 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚73.66549698189647 disk.sda.in‚0! disk.dm-0.out‚1683.5087890625( proc.memory.availableMB‚ 353.40234375* proc.memory.percent‚16.819381713867188 cpu‚25.037803309338607[ÆŸ0ò+ _wandbruntime‚510  _runtime‚510¯W§BÄ&ó¾M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚21141254232576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9775324 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22699019363840 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚21024794860558 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23879384 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5716877 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9775324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22153121456128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1011621770752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚21024554179390 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1008920101902 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1250 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23903584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1032504649216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1011621770752 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9775322 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14104062 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14128260 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9289016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1008696460094 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚549 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1032757447168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22153121456128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22698995423744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23879384 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5716877 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 MADeviceStatsMoni±°?!¤tor.on_train_batch_end/active.large_pool.allocated‚9775324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22153121456128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1011621770752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚21024554179390 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1008920101902 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1250 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 284543488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23903584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1032504649216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1011621770752 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9775322 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14104062 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14128260 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9289016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1008696460094 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚549*{mØZ0õ+ _wandbruntime‚510  _runtime‚510­°YÌPöÊ /loss/FRt1/assignment_loss‚6.120589304943474 2loss/SRqqt4/assignment_loss‚2.6526444699313187 /loss/FRt3/detection_loss‚2.7533003898958364 _step‚43 /loss/FRt2/assignment_loss‚4.273882246017456 2loss/SRqqt2/assignment_loss‚2.9674999973048335 2loss/SRqqt3/assignment_loss‚2.8084150587811187 %loss/total_loss‚72.03431055864188 /loss/FRt4/assignment_loss‚6.152515543831719 0loss/FBt4/assignment_loss‚0.8325273936445062 /loss/FRt1/detection_loss‚2.8072445546181832 epoch‚0 0loss/FBt1/assignment_loss‚0.6721209421753883 1loss/SRqqt1/detection_loss‚2.9262407837729705 /loss/FBt1/detection_loss‚2.3777875555094248 .loss/FBt2/detection_loss‚1.735430932494209 1loss/SRqqt4/detection_loss‚3.2888348972475208 0loss/FRt3/assignment_loss‚5.7952137390772505 1loss/SRqqt1/assignment_loss‚2.883106181496068 .loss/FRt2/detection_loss‚2.246043315759072 0loss/FBt2/assignment_loss‚0.3416378680364384 0loss/FBt3/assignment_loss‚0.4475648690903377 .loss/FRt4/detection_loss‚3.183796611097124 1loss/SRqqt2/detection_loss‚0.8555739459622166 .loss/FBt3/detection_loss‚5.873207921567171 # _timestamp‚1.78841703993957e+09 0loss/SRqqt3/detection_loss‚3.020599890281172 .loss/FBt4/detection_loss‚5.018532146107066 trainer/global_step‚549 _runtime‚ 510.689382014¹ú/ñT÷Î 2loss/SRqqt4/assignment_loss‚2.6526444699313187 /loss/FRt3/detection_loss‚2.7533003898958364 _step‚43 /loss/FRt2/assignment_loss‚4.273882246017456 2loss/SRqqt2/assignment_loss‚2.9674999973048335 2loss/SRqqt3/assignment_loss‚2.8084150587811187 %loss/total_loss‚72.03431055864188 /loss/FRt4/assignment_loss‚6.152515543831719 0loss/FBt4/assignment_loss‚0.8325273936445062 /loss/FRt1/detection_loss‚2.8072445546181832 epoch‚0 0loss/FBt1/assignment_loss‚0.6721209421753883 1loss/SRqqt1/detection_loss‚2.9262407837729705 /loss/FBt1/detection_loss‚2.3777875555094248 .loss/FBt2/detection_loss‚1.735430932494209 1loss/SRqqt4/detection_loss‚3.2888348972475208 0loss/FRt3/assignment_loss‚5.7952137390772505 1loss/SRqqt1/assignment_loss‚2.883106181496068 .loss/FRt2/detection_loss‚2.246043315759072 0loss/FBt2/assignment_loss‚0.3416378680364384 0loss/FBt3/assignment_loss‚0.4475648690903377 .loss/FRt4/detection_loss‚3.183796611097124 1loss/SRqqt2/detection_loss‚0.8555739459622166 .loss/FBt3/detection_loss‚5.873207921567171 # _timestamp‚1.78841703993957e+09 0loss/SRqqt3/detection_loss‚3.020599890281172 .loss/FBt4/detection_loss‚5.018532146107066 trainer/global_step‚549 _runtime‚ 510.689382014 /loss/FRt1/assignment_loss‚6.120589304943474+?¾1ø‚(:„ ž¨äÔßç´x gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.289' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â…B¹ù‚(:Œ ž¨äÔ°»³{ network.recv‚269408 disk.dm-1.in‚0 disk.rbd4.out‚0 proc.memory.rssMB‚ 2755.6875% disk.nvme0c0n1.out‚1684.298828125 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0# disk.nvme0n1.out‚1684.298828125 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.sda.in‚0 disk.sda3.out‚ 19.05078125 disk.rbd1.in‚0 disk.dm-0.in‚4 disk.dm-1.out‚ 19.05078125$ proc.memory.availableMB‚353.3125 network.sent‚754312 disk.nvme0c0n1.in‚4 disk.sda.out‚ 19.05078125 disk.nvme0n1.in‚4$ memory_percent‚97.84355163574219) disk./.usagePercent‚73.66549595786633 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd2.out‚ 32.765625 disk.dm-0.out‚1684.298828125 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd1.out‚ 0.06298828125 disk.rbd2.in‚0* proc.memory.percent‚16.819381713867188 cpu‚25.038591882751213 proc.cpu.threads‚17% disk./.usageGB‚1097.6691246032715 disk.rbd3.out‚0 disk.sda2.in‚0*–Ó˜ú‚(:„ ­¨äÔÌîÌx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.544' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1088 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾’öû‚(:’ ­¨äÔ‡þš{% disk./.usageGB‚1097.6691818237305 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0% disk.nvme0c0n1.out‚1783.287109375 disk.rbd3.out‚0 disk.sda3.out‚ 19.4765625 disk.dm-1.in‚0 disk.dm-1.out‚ 19.4765625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2755.69140625* proc.memory.percent‚16.819405555725098 disk.rbd3.in‚0 disk.sda.out‚ 19.4765625 disk.dm-0.out‚1783.287109375( proc.memory.availableMB‚ 353.56640625 network.recv‚275127 disk.rbd1.in‚0 disk.dm-0.in‚4 disk.nvme0n1.in‚4 disk.sda2.in‚0 disk.rbd2.out‚ 34.30078125 cpu‚25.03891169558673) disk./.usagePercent‚73.66549979797934 disk.nvme0c0n1.in‚4 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0$ memory_percent‚97.84200191497803 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda.in‚0 disk.rbd1.out‚ 0.21142578125# disk.nvme0n1.out‚1783.287109375 network.sent‚793850 disk.rbd4.in‚0 disk.rbd2.in‚0·].#ü‚(:… ¼¨äÔ¸ïx gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.392' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËJ›[%ý‚(:š ¼¨äÔÛæºz disk.rbd0.out‚0 disk.sda2.out‚0$ memory_percent‚97.84119129180908" proc.memory.rssMB‚ 2755.6953125* proc.memory.percent‚16.819429397583008 proc.cpu.threads‚17 network.recv‚278854% disk./.usageGB‚1097.6691246032715 disk.sda.in‚0 disk.sda.out‚ 20.00390625 disk.drbd1777.in‚0 disk.dm-1.out‚ 20.00390625 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚1882.48974609375" disk.dm-0.out‚1882.48974609375 disk.rbd2.out‚ 34.3828125% disk.nvme0n1.out‚1882.48974609375 disk.dm-0.in‚4) disk./.usagePercent‚73.66549595786633 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 network.sent‚799402 disk.sda3.in‚0 disk.sda3.out‚ 20.00390625 disk.rbd1.out‚ 0.21142578125 disk.nvme0n1.in‚4 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 353.69921875 cpu‚25.038835189998018px0þ+ _wandbruntime‚555  _runtime‚555cÛˆÂÿ¼ % lr-AdamW/pg1‚0.000397509416015995 % lr-AdamW/pg2‚0.000397509416015995 trainer/global_step‚599 % _timestamp‚1.7884170848489144e+09 _runtime‚ 555.598899385 _step‚44øeÆ€À _step‚44 % lr-AdamW/pg1‚0.000397509416015995 % lr-AdamW/pg2‚0.000397509416015995 trainer/global_step‚599 % _timestamp‚1.7884170848489144e+09 _runtime‚ 555.598899385,ŠÙ0+ _wandbruntime‚555  _runtime‚555óÅÀ'‚ºO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6224272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299820032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15359413 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24716334480384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10643553 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1098721273302 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 trainer/global_step‚599 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26027140 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚ 555.599855507 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24121742144512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275879936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22893416466902 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 % _timestamp‚1.7884170848496652e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚21794695193600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3892351 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25840770213888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1101910649856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10116623 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3891666 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15383585 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10643555 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1124459673600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24121742144512 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23019813491712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10115936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23019813491712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24121496705024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23019831494656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1124735553536 “ë~i!ƒ¾O XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23019813491712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24121496705024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23019831494656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1124735553536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6224272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299820032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15359413 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24716334480384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10643553 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1098721273302 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 trainer/global_step‚599 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26027140 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚ 555.599855507 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24121742144512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275879936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22893416466902 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 % _timestamp‚1.7884170848496652e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚21794695193600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3892351 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25840770213888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1101910649856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10116623 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3891666 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15383585 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10643555 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1124459673600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24121742144512 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23019813491712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10115936-¶}j.0„+ _wandbruntime‚556  _runtime‚556Z Ê&…ÄM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26070436 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6234625 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26070436 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3898152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚21830997651456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚23058169438720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15384964 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6234627 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24757504176640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚599 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22931310643398 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10661272 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _runtime‚ 556.477246295 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1100330031302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24161672343552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23058151435776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1103520907776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10132777 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25884102724608 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1100553673110 % _timestamp‚1.7884170857267654e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26046236 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24757480236544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1126598547968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287223808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1126335264256 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1103748357632 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1103748357632 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24161917796352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 _step‚46 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1103520907776 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15409162 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23058151435776 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25883815500800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 263283712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23058169438720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3898793¨YÎ&†ÈM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23058151435776 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25883815500800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 263283712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23058169438720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3898793 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26070436 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6234625 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26070436 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3898152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚21830997651456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚23058169438720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15384964 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6234627 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24757504176640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚599 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22931310643398 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10661272 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _runtime‚ 556.477246295 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1100330031302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24161672343552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23058151435776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 326151168 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1103520907776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10132777 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25884102724608 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1100553673110 % _timestamp‚1.7884170857267654e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26046236 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24757480236544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1126598547968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287223808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1126335264256 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1103748357632 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1103748357632 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24161917796352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 _step‚46 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1103520907776 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15409162.`á•0‡+ _wandbruntime‚556  _runtime‚556 ÄÃ2QˆË /loss/FRt3/assignment_loss‚4.900636270642281 0loss/FBt3/assignment_loss‚0.5244251585662434 0loss/FBt4/assignment_loss‚0.3479201558511704 /loss/FBt2/detection_loss‚1.4487829361704812 trainer/global_step‚599 _runtime‚ 556.478295362 _step‚47 2loss/SRqqt2/assignment_loss‚2.5977175550146416 2loss/SRqqt4/assignment_loss‚3.2663216008696447 .loss/FBt4/detection_loss‚6.453641104698181 /loss/FRt2/assignment_loss‚4.905791613790724 .loss/FRt1/detection_loss‚2.214244206668809 % _timestamp‚1.7884170857284453e+09 /loss/FRt4/assignment_loss‚6.258781476454302 2loss/SRqqt3/assignment_loss‚2.0326963737607002 %loss/total_loss‚73.74699621524942 1loss/SRqqt1/assignment_loss‚2.671705823391676 /loss/FRt2/detection_loss‚2.6826133926709494 /loss/SRqqt3/detection_loss‚2.73766465857625 1loss/SRqqt4/detection_loss‚2.5972285908322004 /loss/FRt1/assignment_loss‚5.982822187244892 0loss/FBt1/assignment_loss‚0.8488243111703349 1loss/FBt2/assignment_loss‚0.42358699250726733 .loss/FRt3/detection_loss‚3.338937448337674 1loss/SRqqt1/detection_loss‚3.0581089686602354 1loss/SRqqt2/detection_loss‚1.0071368959865399 /loss/FRt4/detection_loss‚2.7428562171531445 .loss/FBt1/detection_loss‚6.326962965312551 .loss/FBt3/detection_loss‚4.377589310918536 epoch‚0Œg¡¿U‰Ï /loss/FRt3/assignment_loss‚4.900636270642281 0loss/FBt3/assignment_loss‚0.5244251585662434 0loss/FBt4/assignment_loss‚0.3479201558511704 /loss/FBt2/detection_loss‚1.4487829361704812 trainer/global_step‚599 _runtime‚ 556.478295362 _step‚47 2loss/SRqqt2/assignment_loss‚2.5977175550146416 2loss/SRqqt4/assignment_loss‚3.2663216008696447 .loss/FBt4/detection_loss‚6.453641104698181 /loss/FRt2/assignment_loss‚4.905791613790724 .loss/FRt1/detection_loss‚2.214244206668809 % _timestamp‚1.7884170857284453e+09 /loss/FRt4/assignment_loss‚6.258781476454302 2loss/SRqqt3/assignment_loss‚2.0326963737607002 %loss/total_loss‚73.74699621524942 1loss/SRqqt1/assignment_loss‚2.671705823391676 /loss/FRt2/detection_loss‚2.6826133926709494 /loss/SRqqt3/detection_loss‚2.73766465857625 1loss/SRqqt4/detection_loss‚2.5972285908322004 /loss/FRt1/assignment_loss‚5.982822187244892 0loss/FBt1/assignment_loss‚0.8488243111703349 1loss/FBt2/assignment_loss‚0.42358699250726733 .loss/FRt3/detection_loss‚3.338937448337674 1loss/SRqqt1/detection_loss‚3.0581089686602354 1loss/SRqqt2/detection_loss‚1.0071368959865399 /loss/FRt4/detection_loss‚2.7428562171531445 .loss/FBt1/detection_loss‚6.326962965312551 .loss/FBt3/detection_loss‚4.377589310918536 epoch‚0/Å·æ@Š‚(:… ˨äÔÐŽˆx gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RŒóÍ‹‚(:“ ˨äÔèô¶z disk.dm-0.in‚4 disk.rbd2.out‚ 35.921875 disk.sda3.out‚ 20.5703125 network.recv‚283505 disk.rbd1.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 353.65234375" proc.memory.rssMB‚ 2755.6953125* proc.memory.percent‚16.819429397583008 proc.cpu.threads‚17) disk./.usagePercent‚73.66550005398686 disk.nvme0n1.in‚4 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4 disk.rbd3.in‚0" memory_percent‚97.841477394104 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.sda.in‚0% disk.nvme0n1.out‚1981.56396484375 disk.rbd2.in‚0 disk.rbd0.out‚0 network.sent‚805760 disk.sda2.in‚0' disk.nvme0c0n1.out‚1981.56396484375 cpu‚25.03911919793366% disk./.usageGB‚1097.6691856384277 disk.rbd1.out‚ 0.24365234375" disk.dm-0.out‚1981.56396484375 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.sda.out‚ 20.5703125 disk.sda3.in‚0 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 20.57031250‹ÖýŒ‚(:„ Ú¨äԮʊx gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æq³Y"‚(:— Ú¨äÔ µŒ|$ memory_percent‚97.83999919891357$ disk./.usageGB‚1097.669418334961' disk.nvme0c0n1.out‚2080.88330078125% disk.nvme0n1.out‚2080.88330078125 disk.dm-1.in‚0" proc.memory.rssMB‚ 2755.7109375) proc.memory.percent‚16.81952476501465 network.recv‚290293 disk.sda1.out‚0 disk.sda3.out‚ 21.16796875) disk./.usagePercent‚73.66551567044645 disk.rbd2.in‚0 disk.rbd2.out‚ 37.51171875 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.24365234375 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.nvme0c0n1.in‚4 disk.sda.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚4( proc.memory.availableMB‚ 353.89453125 disk.rbd4.in‚0 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 network.sent‚856309 disk.rbd3.in‚0 disk.sda.out‚ 21.16796875 disk.dm-1.out‚ 21.16796875 proc.cpu.threads‚17 disk.rbd4.out‚0 disk.dm-0.in‚4 cpu‚25.0396523372047 disk.rbd0.in‚0 disk.rbd1.in‚0" disk.dm-0.out‚2080.88330078125 disk.sda2.in‚0PKD#Ž‚(:… é¨äÔ’„‘x gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.978' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fxˆ‚(:‘ é¨äÔ¶™Î{ disk.nvme0n1.in‚4# proc.memory.rssMB‚ 2755.71484375 cpu‚25.039940769915436 disk.rbd2.in‚0 disk.drbd1777.out‚0" disk.dm-0.out‚2081.09814453125 network.recv‚295004 disk.rbd3.out‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 353.63671875 disk.drbd1777.in‚0' disk.nvme0c0n1.out‚2081.09814453125 disk.sda1.in‚0 disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0# disk./.usageGB‚1097.66947555542 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 0.306640625 proc.cpu.threads‚17 disk.sda.out‚ 21.703125 disk.dm-0.in‚4$ memory_percent‚97.84157276153564 disk.sda2.in‚0 disk.dm-1.out‚ 21.703125 disk.sda1.out‚0 disk.sda3.out‚ 21.703125) proc.memory.percent‚16.81954860687256 network.sent‚862713) disk./.usagePercent‚73.66551951055948% disk.nvme0n1.out‚2081.09814453125 disk.rbd2.out‚ 37.53515625 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4~ù%Š0+ _wandbruntime‚601  _runtime‚6013aþ‘Ä‘¾ _runtime‚ 601.728506236 _step‚48 & lr-AdamW/pg1‚0.0004306905024947925 & lr-AdamW/pg2‚0.0004306905024947925 trainer/global_step‚649 % _timestamp‚1.7884171309785829e+09Í:È’Â & lr-AdamW/pg1‚0.0004306905024947925 & lr-AdamW/pg2‚0.0004306905024947925 trainer/global_step‚649 % _timestamp‚1.7884171309785829e+09 _runtime‚ 601.728506236 _step‚480Ñä®10“+ _wandbruntime‚601  _runtime‚601¥Ó¦’ ”ºO XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24800172930910 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11529505 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1193809786880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 326151168 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26130538484736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1193582350336 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27993163700224 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24936710694912 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚24799932249950 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10958147 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24936710694912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16640315 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11529503 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6742022 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27993457228800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_byteç~gî.s.large_pool.allocated‚26774819293184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26130538484736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6742020 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4215480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1193809786880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11529505 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10957500 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24936728697856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26774795353088 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23609818086400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1190354844510 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _runtime‚ 601.729626337 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16664487 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26130293045248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1218368347136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24936728697856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4216125 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26130293045248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1193582350336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16640315 _step‚49 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1218637935616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1190131202910 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28169818 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 % _timestamp‚1.7884171309794276e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16664487 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 trainer/global_step‚649 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293528576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269588480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11529503 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24936710694912 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚24799932249950 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10958147 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24936710694912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16640315 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11529503 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6742022 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27993457228800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26774819293184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26130538484736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6742020 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4215480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1193809786880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11529505 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10957500 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24936728697856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26774795353088 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23609818086400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1190354844510 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _runtime‚ 601.729626337 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16664487 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26130293045248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1218368347136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24936728697856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4216125 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26130293045248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1193582350336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16640315 _step‚49 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1218637935616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1190131202910 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28169818 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 % _timestamp‚1.7884171309794276e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16664487 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 trainer/global_step‚649 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293528576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 2695884801r0–+ _wandbruntime‚602  _runtime‚602»VEÊ&—ÄM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11547224 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11547224 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26170714136576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16665866 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10974665 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1195647494656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1195647494656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273769472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24975066641920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24975066641920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16665866 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26170468683776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6752377 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23646120544256 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4221678 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28213088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11547222 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚24838067107406 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 % _timestamp‚1.7884171318426297e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28036497459200 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26815988989440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28237288 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28036199749632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16690064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 297709568 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚24838307788574 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23646103504896 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚610 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚50 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11547222 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24975048638976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 _runtime‚ 602.593099295 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28237288 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26815965049344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16690064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10974053 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4222288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1220234700288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6752375 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1191963602510 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561‚-<$ñ˜ÈM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28237288 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28036199749632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16690064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 297709568 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚24838307788574 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23646103504896 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚610 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚50 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11547222 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24975048638976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 _runtime‚ 602.593099295 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28237288 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26815965049344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16690064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10974053 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4222288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1220234700288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6752375 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1191963602510 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11547224 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 NADeviceStatsMonitor.on_train_bvôRÌÝatch_end/active.large_pool.allocated‚11547224 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26170714136576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16665866 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10974665 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1195647494656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1195647494656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273769472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24975066641920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24975066641920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16665866 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26170468683776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6752377 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23646120544256 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4221678 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28213088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11547222 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚24838067107406 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 % _timestamp‚1.7884171318426297e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28036497459200 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26815988989440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚22¢r)$0™+ _wandbruntime‚602  _runtime‚602zßsAPšÊ /loss/FBt4/detection_loss‚17.034770011901855 0loss/FRt4/assignment_loss‚6.1660934044764595 1loss/FBt3/assignment_loss‚0.24157920802528376 0loss/SRqqt3/detection_loss‚2.608441154746448 -loss/FBt1/detection_loss‚2.32846136567344 /loss/FRt3/assignment_loss‚5.644205429615119 1loss/FBt2/assignment_loss‚0.27908088122222807 1loss/SRqqt4/detection_loss‚2.5653847083449364 .loss/FBt3/detection_loss‚6.567204481363296 _step‚51 1loss/SRqqt1/assignment_loss‚2.889688112519004 /loss/FRt1/detection_loss‚2.6392138557301625 $loss/total_loss‚81.1052824385929 trainer/global_step‚649 1loss/FBt4/assignment_loss‚0.15566717088222504 .loss/FRt2/detection_loss‚1.993908642463281 0loss/SRqqt2/detection_loss‚1.002481928203515 /loss/FBt2/detection_loss‚1.5376034225919284 2loss/SRqqt4/assignment_loss‚2.4304270282387734 .loss/FRt4/detection_loss‚2.715513880436237 epoch‚0 % _timestamp‚1.7884171318442492e+09 _runtime‚ 602.594087517 /loss/FRt1/assignment_loss‚6.312827576531304 /loss/FRt2/assignment_loss‚4.625293661171282 2loss/SRqqt3/assignment_loss‚2.5270195603370667 1loss/SRqqt2/assignment_loss‚2.577978931194128 0loss/FBt1/assignment_loss‚0.6076195961680594 .loss/FRt3/detection_loss‚2.853976228298285 0loss/SRqqt1/detection_loss‚2.800842198458585ª/T›Î /loss/FBt4/detection_loss‚17.034770011901855 0loss/FRt4/assignment_loss‚6.1660934044764595 1loss/FBt3/assignment_loss‚0.24157920802528376 0loss/SRqqt3/detection_loss‚2.608441154746448 -loss/FBt1/detection_loss‚2.32846136567344 /loss/FRt3/assignment_loss‚5.644205429615119 1loss/FBt2/assignment_loss‚0.27908088122222807 1loss/SRqqt4/detection_loss‚2.5653847083449364 .loss/FBt3/detection_loss‚6.567204481363296 _step‚51 1loss/SRqqt1/assignment_loss‚2.889688112519004 /loss/FRt1/detection_loss‚2.6392138557301625 $loss/total_loss‚81.1052824385929 trainer/global_step‚649 1loss/FBt4/assignment_loss‚0.15566717088222504 .loss/FRt2/detection_loss‚1.993908642463281 0loss/SRqqt2/detection_loss‚1.002481928203515 /loss/FBt2/detection_loss‚1.5376034225919284 2loss/SRqqt4/assignment_loss‚2.4304270282387734 .loss/FRt4/detection_loss‚2.715513880436237 epoch‚0 % _timestamp‚1.7884171318442492e+09 _runtime‚ 602.594087517 /loss/FRt1/assignment_loss‚6.312827576531304 /loss/FRt2/assignment_loss‚4.625293661171282 2loss/SRqqt3/assignment_loss‚2.5270195603370667 1loss/SRqqt2/assignment_loss‚2.577978931194128 0loss/FBt1/assignment_loss‚0.6076195961680594 .loss/FRt3/detection_loss‚2.853976228298285 0loss/SRqqt1/detection_loss‚2.8008421984585853í- M›œ‚(: ø¨äÔØþx gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05~d‚(:Œ ø¨äÔÜòÀz$ memory_percent‚97.84233570098877 cpu‚25.040265433290248 proc.cpu.threads‚17 network.sent‚868457 network.recv‚299227 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 22.26953125 disk.rbd1.out‚ 0.306640625) proc.memory.percent‚16.81957244873047 disk.rbd4.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 22.26953125 proc.memory.rssMB‚ 2755.71875 disk.nvme0c0n1.in‚4 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚2180.861328125# disk.nvme0n1.out‚2180.861328125 disk.dm-0.out‚2180.861328125 disk.rbd2.in‚0 disk.rbd2.out‚39.1875( proc.memory.availableMB‚ 353.51171875) disk./.usagePercent‚73.66553077489098% disk./.usageGB‚1097.6696434020996 disk.nvme0n1.in‚4 disk.sda2.out‚0 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 22.26953125 disk.rbd1.in‚0 disk.dm-0.in‚4 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-1.in‚0[è Üž‚(:… ‡©äÔ‚Š¤x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.508' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7016 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ““¬Ÿ‚(:Ž ‡©äÔüªê{ disk.rbd1.in‚0" disk.dm-0.out‚2280.08154296875 disk.rbd2.out‚ 39.21484375 disk.sda1.in‚0$ proc.memory.availableMB‚353.5625 disk.rbd1.out‚ 0.36962890625 disk.sda1.out‚0% disk./.usageGB‚1097.6696434020996 disk.sda2.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 network.recv‚304653 disk.nvme0n1.in‚4 disk.rbd4.in‚0 disk.nvme0c0n1.in‚4 disk.sda.out‚ 22.796875 disk.sda3.out‚ 22.796875 cpu‚25.040383391258178 network.sent‚918028) disk./.usagePercent‚73.66553077489098 disk.dm-1.out‚ 22.796875 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.sda.in‚0 disk.dm-0.in‚4 disk.sda3.in‚0 proc.memory.rssMB‚ 2755.71875) proc.memory.percent‚16.81957244873047% disk.nvme0n1.out‚2280.08154296875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚2280.08154296875 disk.rbd3.out‚0$ memory_percent‚97.84202575683594 proc.cpu.threads‚17¨Žew ‚(:… –©äÔ à©x gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wé{ó¡‚(:‡ –©äÔûÇÃ| network.sent‚925999) disk./.usagePercent‚73.66553231093619 disk.rbd4.out‚0 disk.nvme0n1.out‚ 2379.46875 disk.dm-0.in‚4 disk.rbd2.in‚0 disk.rbd2.out‚ 41.0546875 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2755.72265625 disk.nvme0c0n1.in‚4! disk.nvme0c0n1.out‚ 2379.46875 disk.sda.out‚ 23.19140625 disk.nvme0n1.in‚4 disk.rbd0.out‚0 cpu‚25.040543793404215 network.recv‚308750 disk.sda1.out‚0 disk.sda3.out‚ 23.19140625 disk.sda2.in‚0 disk.dm-1.out‚ 23.19140625) proc.memory.percent‚16.81959629058838 disk.rbd3.in‚0 disk.sda.in‚0 disk.rbd1.in‚0 disk.dm-0.out‚ 2379.46875 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.rbd1.out‚ 0.36962890625 disk.drbd1777.out‚0' proc.memory.availableMB‚ 353.5546875% disk./.usageGB‚1097.6696662902832 disk.rbd4.in‚0$ memory_percent‚97.84207344055176ð”‡e0¢+ _wandbruntime‚647  _runtime‚647-ò†µÃ£½ _step‚52 & lr-AdamW/pg1‚0.0004638715889735901 & lr-AdamW/pg2‚0.0004638715889735901 trainer/global_step‚699 $ _timestamp‚1.788417176976403e+09 _runtime‚ 647.726343089­¬!JǤÁ & lr-AdamW/pg2‚0.0004638715889735901 trainer/global_step‚699 $ _timestamp‚1.788417176976403e+09 _runtime‚ 647.726343089 _step‚52 & lr-AdamW/pg1‚0.00046387158897359014Š¢™ 0¥+ _wandbruntime‚647  _runtime‚647:ÍnÀ'¦ºO XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚26706929394918 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30145531550720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30145246410752 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 647.727318226 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25424923939840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1281764774118 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30336670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚26853607898112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26853607898112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1312227444736 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28139089385472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1285481487360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30145531550720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30145246410752 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 647.727318226 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25424923939840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1281764774118 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30336670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚26853607898112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26853607898112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1312227444736 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28139089385472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1285481487360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚54 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17970966 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12433172 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11816524 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26745064252582 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26744823571414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12433174 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4547061 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30404140 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17970966 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17946768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30188283269120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1314107176448 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.788417177844059e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 648.594759582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26891945842176 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1313833406976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26891963845120 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11817188 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 328248320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28874473802240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12433172 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7270125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25461243437056 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30379940 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30188580978688f=xÉÍ&ªÇM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26891945842176 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1313833406976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26891963845120 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11817188 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 328248320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28874473802240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12433172 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7270125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25461243437056 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30379940 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30188580978688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1283820815526 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28179510476800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 297709568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30379940 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1287319181824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25461226397696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚54 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17970966 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12433172 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11816524 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26745064252582 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26744823571414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12433174 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4547061 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30404140 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17970966 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17946768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30188283269120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1314107176448 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.788417177844059e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 648.5947595826`Cô"0«+ _wandbruntime‚648  _runtime‚648Ù è‡P¬Ê 1loss/SRqqt2/assignment_loss‚2.526690536075168 0loss/FBt4/assignment_loss‚0.4990997239947319 .loss/FRt1/detection_loss‚2.291066383728041 /loss/FRt3/detection_loss‚2.9776056294932083 2loss/SRqqt1/assignment_loss‚2.5310728549957275 1loss/FBt2/assignment_loss‚0.24118868383097972 trainer/global_step‚699 _step‚55 /loss/FRt2/assignment_loss‚4.084277461315024 1loss/SRqqt3/assignment_loss‚2.875652749197824 -loss/FBt4/detection_loss‚8.57020029425621 /loss/FRt1/assignment_loss‚5.919957564427302 1loss/SRqqt1/detection_loss‚3.7824259057641028 /loss/FBt2/detection_loss‚1.2719102664145105 % _timestamp‚1.7884171778460119e+09 epoch‚0 /loss/FRt3/assignment_loss‚5.257029954124899 2loss/SRqqt4/assignment_loss‚2.9853563838534884 0loss/SRqqt3/detection_loss‚2.373652126959392 %loss/total_loss‚87.43182659868845 _runtime‚ 648.595855988 0loss/FBt1/assignment_loss‚0.6422337111625893 0loss/FBt3/assignment_loss‚0.5687421237428983 .loss/FRt2/detection_loss‚2.263666282440054 1loss/SRqqt2/detection_loss‚0.9270304834018602 0loss/SRqqt4/detection_loss‚2.797039884659979 .loss/FBt1/detection_loss‚4.143565822392702 .loss/FBt3/detection_loss‚19.46775804956754 0loss/FRt4/assignment_loss‚5.8930115302403765 /loss/FRt4/detection_loss‚2.5415921926498415»®T­Î 1loss/SRqqt3/assignment_loss‚2.875652749197824 -loss/FBt4/detection_loss‚8.57020029425621 /loss/FRt1/assignment_loss‚5.919957564427302 1loss/SRqqt1/detection_loss‚3.7824259057641028 /loss/FBt2/detection_loss‚1.2719102664145105 % _timestamp‚1.7884171778460119e+09 epoch‚0 /loss/FRt3/assignment_loss‚5.257029954124899 2loss/SRqqt4/assignment_loss‚2.9853563838534884 0loss/SRqqt3/detection_loss‚2.373652126959392 %loss/total_loss‚87.43182659868845 _runtime‚ 648.595855988 0loss/FBt1/assignment_loss‚0.6422337111625893 0loss/FBt3/assignment_loss‚0.5687421237428983 .loss/FRt2/detection_loss‚2.263666282440054 1loss/SRqqt2/detection_loss‚0.9270304834018602 0loss/SRqqt4/detection_loss‚2.797039884659979 .loss/FBt1/detection_loss‚4.143565822392702 .loss/FBt3/detection_loss‚19.46775804956754 0loss/FRt4/assignment_loss‚5.8930115302403765 /loss/FRt4/detection_loss‚2.5415921926498415 1loss/SRqqt2/assignment_loss‚2.526690536075168 0loss/FBt4/assignment_loss‚0.4990997239947319 .loss/FRt1/detection_loss‚2.291066383728041 /loss/FRt3/detection_loss‚2.9776056294932083 2loss/SRqqt1/assignment_loss‚2.5310728549957275 1loss/FBt2/assignment_loss‚0.24118868383097972 trainer/global_step‚699 _step‚55 /loss/FRt2/assignment_loss‚4.0842774613150247m7Çô‹®‚(:€ ¥©äÔ­Ç™x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rʯ‚(:Ž ¥©äÔóéþz disk.rbd3.in‚0* proc.memory.percent‚16.820144653320312 disk.sda3.out‚ 23.609375 disk.rbd4.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚313435 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚4" disk.dm-0.out‚2379.78564453125 disk.sda.out‚ 23.609375 disk.nvme0n1.in‚4% disk.nvme0n1.out‚2379.78564453125 disk.dm-0.in‚4 disk.sda2.in‚0 cpu‚25.040866980723454 disk.sda2.out‚0 disk.rbd3.out‚0 proc.memory.rssMB‚ 2755.8125) disk./.usagePercent‚73.66550799022043 disk.sda.in‚0 disk.dm-1.out‚ 23.609375 disk.rbd2.out‚ 41.0546875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0$ memory_percent‚97.84650802612305' disk.nvme0c0n1.out‚2379.78564453125 disk.rbd4.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.36962890625 disk.dm-1.in‚0 disk.drbd1777.in‚0& proc.memory.availableMB‚ 352.828125 network.sent‚930140$ disk./.usageGB‚1097.669303894043B;ó}°‚(:… ´©äÔƒàx gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.149' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦³Ò4±‚(: ´©äÔ“ªÑz$ memory_percent‚97.84507751464844 disk.nvme0n1.in‚4$ disk.nvme0n1.out‚2479.0634765625 disk.dm-0.in‚4 disk.rbd1.in‚0$ proc.memory.availableMB‚353.0625 proc.memory.rssMB‚ 2755.8125 network.sent‚979319% disk./.usageGB‚1097.6698570251465 disk.sda2.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚318566( disk./.usagePercent‚73.6655451113129 disk.rbd2.out‚ 42.79296875 disk.drbd1777.in‚0 disk.rbd4.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 24.21484375 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4 disk.sda3.in‚0 disk.sda3.out‚ 24.21484375 cpu‚25.041088550505997* proc.memory.percent‚16.820144653320312 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 24.21484375 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 0.36962890625! disk.dm-0.out‚2479.0634765625 disk.rbd2.in‚0& disk.nvme0c0n1.out‚2479.0634765625 disk.rbd4.out‚0 disk.sda1.out‚0W,ü²‚(:… éäÔ˜ƒªx gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.566' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5132 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M Ú³‚(:Š éäÔ­ÑÈz network.sent‚985235 disk.drbd1777.out‚0 network.recv‚323177 disk.sda.in‚0 disk.sda.out‚ 24.671875& disk.nvme0c0n1.out‚2578.0908203125$ proc.memory.availableMB‚352.5625* proc.memory.percent‚16.820144653320312 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4 disk.rbd3.in‚0 proc.memory.rssMB‚ 2755.8125 cpu‚25.041279821454822 disk.rbd1.in‚0 disk.rbd1.out‚ 0.36962890625 disk.dm-0.in‚4 disk.sda2.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚4! disk.dm-0.out‚2578.0908203125 disk.dm-1.out‚ 24.671875 disk.rbd4.in‚0% disk./.usageGB‚1097.6696586608887$ disk.nvme0n1.out‚2578.0908203125 disk.rbd2.out‚ 44.3046875 disk.sda1.out‚0) disk./.usagePercent‚73.66553179892111 disk.sda3.out‚ 24.671875 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0$ memory_percent‚97.84812927246094 proc.cpu.threads‚17á±?0´+ _wandbruntime‚693  _runtime‚693þ‹Äµ¾ _step‚56 & lr-AdamW/pg1‚0.0004970526754523877 & lr-AdamW/pg2‚0.0004970526754523877 trainer/global_step‚749 % _timestamp‚1.7884172224759307e+09 _runtime‚ 693.225918316«oØ$ȶ & lr-AdamW/pg1‚0.0004970526754523877 & lr-AdamW/pg2‚0.0004970526754523877 trainer/global_step‚749 % _timestamp‚1.7884172224759307e+09 _runtime‚ 693.225918316 _step‚568N¬´¾0·+ _wandbruntime‚693  _runtime‚693©üìÓ|¸ºO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19202119 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28770523104256 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4865019 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1377608060928 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1405876091904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27240063872000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30147885725696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13301405 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19226291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28770523104256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27240046832640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12642541 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30147885725696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28770505101312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30148131165184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12641888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32297369384960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 328248320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28613445177966 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1377608060928 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27240046832640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12642541 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30147885725696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28770505101312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30148131165184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12641888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32297369384960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 328248320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28613445177966 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1377608060928 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30147885725696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13301405 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚192262919¼Å(€0º+ _wandbruntime‚694  _runtime‚694µ®zëÊ&»ÄM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30188306817024 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27276366329856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1375454386734 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12659066 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7787875 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚58 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1379445768704 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277963776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13319124 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19251868 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1379218318848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32546792 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27276349290496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1379445768704 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13319122 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28808843045376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19227670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4871829 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1375230744926 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 % _timestamp‚1.7884172233273177e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32570992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32340432139264 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30932934674944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19227670 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 286640640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30188061364224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28808861048320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1407775428096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301903872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28651820716590 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28808843045376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 328248320 trainer/global_step‚749 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4871191 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28651580035422 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7787877 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32340734043136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1407497464320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Þ¨ñ—Û¼ÈM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301903872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28651820716590 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28808843045376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 328248320 trainer/global_step‚749 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4871191 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28651580035422 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7787877 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32340734043136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1407497464320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30188306817024 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27276366329856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1375454386734 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12659066 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7787875 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚58 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1379445768704 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277963776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13319124 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19251868 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1379218318848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32546792 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27276349290496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1379445768704 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13319122 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28808843045376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19227670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4871829 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1375230744926 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 % _timestamp‚1.7884172233273177e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32570992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32340432139264 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30932934674944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19227670 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 286640640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30188061364224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28808861048320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1407775428096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824:Æó6E0½+ _wandbruntime‚694  _runtime‚694MÒúT¾Î 2loss/SRqqt1/assignment_loss‚2.7772776713738074 .loss/FBt2/detection_loss‚1.172489766544406 /loss/FRt2/assignment_loss‚4.126836422463538 /loss/FRt3/detection_loss‚2.6832557015302703 /loss/SRqqt3/detection_loss‚2.51945569117864 /loss/FBt3/detection_loss‚5.0781833827495575 /loss/FBt4/detection_loss‚2.7673123966563833 /loss/FRt3/assignment_loss‚5.397866435167266 0loss/FBt1/assignment_loss‚0.5163297194521874 1loss/FBt2/assignment_loss‚0.23687521626444336 -loss/FBt1/detection_loss‚6.42674472657117 trainer/global_step‚749 $ _timestamp‚1.788417223329249e+09 2loss/SRqqt3/assignment_loss‚3.0886611649484346 2loss/SRqqt4/assignment_loss‚2.8354841470718384 1loss/SRqqt4/detection_loss‚2.1323871605775575 1loss/SRqqt2/detection_loss‚0.8688980659798664 epoch‚0 _step‚59 0loss/FRt4/assignment_loss‚5.4162905641964505 0loss/FBt3/assignment_loss‚0.7420389434118988 1loss/SRqqt1/detection_loss‚2.7877819331792684 /loss/FRt1/assignment_loss‚5.880739789260061 2loss/SRqqt2/assignment_loss‚2.7356350445279887 0loss/FBt4/assignment_loss‚0.5886619851755147 /loss/FRt2/detection_loss‚1.9431526153020455 /loss/FRt4/detection_loss‚2.4666746812207356 %loss/total_loss‚67.89631570171476 _runtime‚ 694.079108025 /loss/FRt1/detection_loss‚2.7072824769114194K^QmX¿Ò epoch‚0 _step‚59 0loss/FRt4/assignment_loss‚5.4162905641964505 0loss/FBt3/assignment_loss‚0.7420389434118988 1loss/SRqqt1/detection_loss‚2.7877819331792684 /loss/FRt1/assignment_loss‚5.880739789260061 2loss/SRqqt2/assignment_loss‚2.7356350445279887 0loss/FBt4/assignment_loss‚0.5886619851755147 /loss/FRt2/detection_loss‚1.9431526153020455 /loss/FRt4/detection_loss‚2.4666746812207356 %loss/total_loss‚67.89631570171476 _runtime‚ 694.079108025 /loss/FRt1/detection_loss‚2.7072824769114194 2loss/SRqqt1/assignment_loss‚2.7772776713738074 .loss/FBt2/detection_loss‚1.172489766544406 /loss/FRt2/assignment_loss‚4.126836422463538 /loss/FRt3/detection_loss‚2.6832557015302703 /loss/SRqqt3/detection_loss‚2.51945569117864 /loss/FBt3/detection_loss‚5.0781833827495575 /loss/FBt4/detection_loss‚2.7673123966563833 /loss/FRt3/assignment_loss‚5.397866435167266 0loss/FBt1/assignment_loss‚0.5163297194521874 1loss/FBt2/assignment_loss‚0.23687521626444336 -loss/FBt1/detection_loss‚6.42674472657117 trainer/global_step‚749 $ _timestamp‚1.788417223329249e+09 2loss/SRqqt3/assignment_loss‚3.0886611649484346 2loss/SRqqt4/assignment_loss‚2.8354841470718384 1loss/SRqqt4/detection_loss‚2.1323871605775575 1loss/SRqqt2/detection_loss‚0.8688980659798664;R6ÒÕšÀ‚(: Ò©äÔžð¸x gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.579' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.915799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉzÁ‚(:“ Ò©äÔêÚ£{ disk.rbd1.in‚0! disk.dm-0.out‚2677.6982421875& proc.memory.availableMB‚ 352.453125 cpu‚25.041066424802487 disk.nvme0c0n1.in‚4 disk.sda.out‚ 25.29296875 disk.sda3.in‚0 disk.rbd1.out‚ 0.36962890625$ disk.nvme0n1.out‚2677.6982421875 disk.sda2.out‚0 disk.dm-1.out‚ 25.29296875 proc.memory.rssMB‚ 2755.8125 network.sent‚991064% disk./.usageGB‚1097.6698760986328 disk.sda1.in‚0 disk.nvme0n1.in‚4 disk.rbd2.in‚0 disk.drbd1777.in‚0* proc.memory.percent‚16.820144653320312 network.recv‚327165 disk.rbd2.out‚ 44.47265625 disk.rbd0.out‚0$ memory_percent‚97.84879684448242 disk.sda3.out‚ 25.29296875) disk./.usagePercent‚73.66554639135057 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-0.in‚4 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 proc.cpu.threads‚17& disk.nvme0c0n1.out‚2677.6982421875 disk.sda1.out‚0 disk.dm-1.in‚0 bÞ`‚(:… á©äÔÛÏ“x gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.322' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eâqqÂ(:’ á©äÔ©Öõz disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚4) disk./.usagePercent‚73.66554639135057% disk./.usageGB‚1097.6698760986328 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0& disk.nvme0c0n1.out‚2677.8974609375" proc.memory.availableMB‚352.75 proc.memory.rssMB‚ 2755.8125$ disk.nvme0n1.out‚2677.8974609375 network.recv‚332163! disk.dm-0.out‚2677.8974609375 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 0.36962890625 disk.dm-0.in‚4* proc.memory.percent‚16.820144653320312 cpu‚25.04160624503547 proc.cpu.threads‚17 disk.dm-1.out‚ 25.80078125 network.sent‚ 1.040235e+06 disk.rbd3.in‚0 disk.sda3.out‚ 25.80078125 disk.sda2.out‚0 disk.nvme0c0n1.in‚4 disk.rbd2.out‚ 45.984375 disk.rbd4.out‚0 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0$ memory_percent‚97.84698486328125 disk.sda.out‚ 25.80078125 disk.rbd4.in‚0M.;³Ä‚(:… ð©äÔ‚Úœx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œq·&#Å‚(:˜ ð©äÔó‡£z! disk.dm-0.out‚2777.0556640625 disk.sda2.in‚0 disk.dm-1.out‚ 26.29296875 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.36962890625 disk.drbd1777.out‚0* proc.memory.percent‚16.820144653320312 cpu‚25.041815514252512 proc.cpu.threads‚17' proc.memory.availableMB‚ 352.9453125 network.sent‚ 1.045429e+06 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.dm-0.in‚4 disk.rbd2.in‚0 disk.nvme0c0n1.in‚4 proc.memory.rssMB‚ 2755.8125 disk.sda3.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0& disk.nvme0c0n1.out‚2777.0556640625 disk.sda1.out‚0 disk.sda.out‚ 26.29296875( disk./.usagePercent‚73.6655451113129% disk./.usageGB‚1097.6698570251465 disk.sda3.out‚ 26.29296875$ disk.nvme0n1.out‚2777.0556640625 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd3.out‚0 network.recv‚335472 disk.nvme0n1.in‚4 disk.rbd2.out‚ 46.2109375 disk.drbd1777.in‚0 disk.sda.in‚0$ memory_percent‚97.84579277038574ùò>0Æ+ _wandbruntime‚738  _runtime‚738©Ú*øÄǾ _step‚60 & lr-AdamW/pg1‚0.0005302337619311853 & lr-AdamW/pg2‚0.0005302337619311853 trainer/global_step‚799 % _timestamp‚1.7884172682167904e+09 _runtime‚ 738.966738024¾%#ÈÈ & lr-AdamW/pg1‚0.0005302337619311853 & lr-AdamW/pg2‚0.0005302337619311853 trainer/global_step‚799 % _timestamp‚1.7884172682167904e+09 _runtime‚ 738.966738024 _step‚60<)zÆ0É+ _wandbruntime‚738  _runtime‚738—±w7¾'ʸO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8295272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 328248320 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34694548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1499322741248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30687420307456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚32156927505408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20483021 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1469279761408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293528576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29055186764800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29055169725440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 _runtime‚ 738.967722308 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32156682065920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚32950249791488 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1469279761408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1465255558134 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20507193 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1469507197952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5189483 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32156682065920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8295270 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34449866061312 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30687420307456 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32156927505408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14187355 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚675 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269588480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20507193 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1465031916534 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1469507197952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚32950273731584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _step‚61 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34449572532736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30520442322934 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5190156 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920_w­ÂÂ'˼O UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 _runtime‚ 738.967722308 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32156682065920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚32950249791488 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1469279761408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1465255558134 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20507193 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1469507197952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5189483 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32156682065920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8295270 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34449866061312 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30687420307456 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32156927505408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14187355 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚675 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269588480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20507193 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1465031916534 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1469507197952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚32950273731584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _step‚61 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34449572532736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30520442322934 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5190156 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8295272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 328248320 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34694548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1499322741248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30687420307456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚32156927505408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20483021 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1469279761408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293528576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29055186764800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29055169725440=Wêϱ0Ì+ _wandbruntime‚739  _runtime‚739H]y‰ÿ ÍÄM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34737844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8305627 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20532770 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/²Ó|Ëoversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1471117455872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚32196857704448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30725740248576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1501193540608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20508572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5196839 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1471344905728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13502466 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29091472183296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1467087957942 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884172690677164e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚32197103157248 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34737844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14205074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29091489222656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30558577180598 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20532770 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5196197 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34492908640768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 295612416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚32991443427840 _step‚62 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14205072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13501822 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 271672320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299Kå4}Î&ÎÈM _step‚62 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14205072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13501822 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 271672320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34737844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8305627 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20532770 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1471117455872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚32196857704448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30725740248576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1501193540608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20508572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5196839 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1471344905728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13502466 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29091472183296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1467087957942 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884172690677164e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚32197103157248 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34737844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14205074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29091489222656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30558577180598 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20532770 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5196197 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34492908640768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 295612416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚32991443427840>ø÷D 0Ï+ _wandbruntime‚739  _runtime‚739û£IƒUÐÏ 2loss/SRqqt1/assignment_loss‚2.4386819216512863 /loss/FRt1/detection_loss‚2.6508567355028014 /loss/FRt3/detection_loss‚2.5299415821614475 1loss/SRqqt4/detection_loss‚2.4300220385193825 /loss/FRt4/assignment_loss‚5.737087033011696 1loss/FBt2/assignment_loss‚0.40446986970139254 0loss/FBt4/assignment_loss‚0.8455894745886325 /loss/FBt2/detection_loss‚0.6059191257096408 % _timestamp‚1.7884172690701876e+09 /loss/FRt1/assignment_loss‚5.819194677399426 0loss/FBt3/assignment_loss‚0.7264247962403131 .loss/FRt4/detection_loss‚2.506766673290368 .loss/FBt3/detection_loss‚4.179795094898769 trainer/global_step‚799 _runtime‚ 739.820043705 _step‚63 /loss/FRt2/assignment_loss‚3.371906288770529 epoch‚0 1loss/SRqqt3/assignment_loss‚2.402211863133642 0loss/FBt1/assignment_loss‚0.7551881683584725 1loss/SRqqt1/detection_loss‚2.7458218124604996 /loss/FRt2/detection_loss‚1.9161596059799195 /loss/FBt4/detection_loss‚3.0394012093544007 2loss/SRqqt2/assignment_loss‚2.6842142946190304 1loss/SRqqt2/detection_loss‚1.1077707403132484 %loss/total_loss‚62.75918062118307 /loss/FRt3/assignment_loss‚5.032039129215738 2loss/SRqqt4/assignment_loss‚2.5661641247570515 1loss/SRqqt3/detection_loss‚2.4624484040670924 .loss/FBt1/detection_loss‚3.801105957478285w—ŠYÑÓ /loss/FRt4/assignment_loss‚5.737087033011696 1loss/FBt2/assignment_loss‚0.40446986970139254 0loss/FBt4/assignment_loss‚0.8455894745886325 /loss/FBt2/detection_loss‚0.6059191257096408 % _timestamp‚1.7884172690701876e+09 /loss/FRt1/assignment_loss‚5.819194677399426 0loss/FBt3/assignment_loss‚0.7264247962403131 .loss/FRt4/detection_loss‚2.506766673290368 .loss/FBt3/detection_loss‚4.179795094898769 trainer/global_step‚799 _runtime‚ 739.820043705 _step‚63 /loss/FRt2/assignment_loss‚3.371906288770529 epoch‚0 1loss/SRqqt3/assignment_loss‚2.402211863133642 0loss/FBt1/assignment_loss‚0.7551881683584725 1loss/SRqqt1/detection_loss‚2.7458218124604996 /loss/FRt2/detection_loss‚1.9161596059799195 /loss/FBt4/detection_loss‚3.0394012093544007 2loss/SRqqt2/assignment_loss‚2.6842142946190304 1loss/SRqqt2/detection_loss‚1.1077707403132484 %loss/total_loss‚62.75918062118307 /loss/FRt3/assignment_loss‚5.032039129215738 2loss/SRqqt4/assignment_loss‚2.5661641247570515 1loss/SRqqt3/detection_loss‚2.4624484040670924 .loss/FBt1/detection_loss‚3.801105957478285 2loss/SRqqt1/assignment_loss‚2.4386819216512863 /loss/FRt1/detection_loss‚2.6508567355028014 /loss/FRt3/detection_loss‚2.5299415821614475 1loss/SRqqt4/detection_loss‚2.4300220385193825?¶ ™^Ò‚(:„ ÿ©äÔ¥‚x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.541' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5082 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–ýHÓ‚(:“ ÿ©äÔ¶„…{ disk.sda.out‚ 26.984375) disk./.usagePercent‚73.66554639135057$ disk.nvme0n1.out‚2876.1103515625 disk.sda2.out‚0 disk.dm-1.out‚ 26.984375 disk.rbd0.out‚0' proc.memory.availableMB‚ 352.8828125% disk./.usageGB‚1097.6698760986328 disk.rbd4.in‚0 disk.nvme0c0n1.in‚4 disk.sda1.out‚0 disk.sda3.out‚ 26.984375 disk.rbd1.in‚0# memory_percent‚97.8461742401123 cpu‚25.042244500526973 disk.nvme0n1.in‚4! disk.dm-0.out‚2876.1103515625 disk.rbd1.out‚ 0.36962890625 network.sent‚ 1.051404e+06 disk.drbd1777.out‚0 disk.rbd4.out‚0 proc.memory.rssMB‚ 2755.8125* proc.memory.percent‚16.820144653320312 disk.dm-0.in‚4 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda.in‚0 network.recv‚339943 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd2.out‚ 47.72265625 disk.drbd1777.in‚0& disk.nvme0c0n1.out‚2876.1103515625GŽÔ‚(:… ŽªäÔéùx gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.584' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Er€Ï Õ‚(:‚ ŽªäÔÚµõz disk.rbd4.in‚0 network.sent‚ 1.100791e+06% disk./.usageGB‚1097.6693725585938 disk.sda2.in‚0( proc.memory.availableMB‚ 352.86328125 cpu‚25.04208331463377 disk.rbd2.in‚0 disk.rbd1.in‚0" disk.nvme0n1.out‚ 2876.59765625 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda.out‚27.75$ disk.nvme0c0n1.out‚ 2876.59765625 proc.cpu.threads‚17 disk.rbd2.out‚ 47.89453125 disk.drbd1777.out‚0 proc.memory.rssMB‚ 2755.8125 disk.dm-1.out‚27.75 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚4 disk.dm-0.out‚ 2876.59765625 disk.sda2.out‚0$ memory_percent‚97.84629344940186 disk.sda3.in‚0 disk.rbd1.out‚ 0.36962890625 network.recv‚345011 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚4* proc.memory.percent‚16.820144653320312) disk./.usagePercent‚73.66551259835605 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4 disk.sda3.out‚27.75*à¤Ö‚(:… ªäÔ¿Ò•x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.867' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9734 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é@É %ׂ(:š ªäÔØÑÊ{ disk.rbd4.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0$ memory_percent‚97.84770011901855 network.sent‚ 1.107137e+06 disk.rbd1.in‚0% disk.nvme0n1.out‚2975.66259765625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚4 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4 disk.dm-1.out‚ 28.37109375 disk.rbd2.out‚49.40625 disk.rbd0.in‚0' proc.memory.availableMB‚ 352.6328125 network.recv‚349840% disk./.usageGB‚1097.6694145202637 disk.sda3.out‚ 28.37109375 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0* proc.memory.percent‚16.820144653320312 proc.cpu.threads‚17 disk.dm-0.in‚4 disk.drbd1777.out‚0 disk.sda.out‚ 28.37109375) disk./.usagePercent‚73.66551541443891 disk.rbd1.out‚ 0.36962890625" disk.dm-0.out‚2975.66259765625 disk.drbd1777.in‚0' disk.nvme0c0n1.out‚2975.66259765625 disk.sda.in‚0 proc.memory.rssMB‚ 2755.8125 disk.sda2.out‚0 cpu‚25.042188172042145¦Æcv0Ø+ _wandbruntime‚784  _runtime‚784K[ÄÙ¾ _runtime‚ 784.892127869 _step‚64 & lr-AdamW/pg1‚0.0005634148484099828 & lr-AdamW/pg2‚0.0005634148484099828 trainer/global_step‚849 % _timestamp‚1.7884173141421912e+09fKÈÚ % _timestamp‚1.7884173141421912e+09 _runtime‚ 784.892127869 _step‚64 & lr-AdamW/pg1‚0.0005634148484099828 & lr-AdamW/pg2‚0.0005634148484099828 trainer/global_step‚849@ ÛèÍ0Û+ _wandbruntime‚784  _runtime‚784xSþ .ܹO _runtime‚ 784.89321586 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32604299507712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36602058030080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 295625728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34165478406144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32604317510656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271685632 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1593027800064 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32427198786942 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21788095 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36837226 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21763923 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5514216 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15073305 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884173141429741e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5514890 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1561406334976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1561178898432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8813022 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30870309657600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1556665487742 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14327912 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1556889129342 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36861400 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚34165478406144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21788095 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0€Ú\×Ã'ݽO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1593027800064 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32427198786942 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21788095 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36837226 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21763923 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5514216 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15073305 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884173141429741e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5514890 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1561406334976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1561178898432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8813022 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30870309657600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1556665487742 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14327912 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1556889129342 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36861400 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚34165478406144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21788095 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 784.89321586 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32604299507712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36602058030080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 295625728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34165478406144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32604317510656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271685632 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624AwKáz0Þ+ _wandbruntime‚785  _runtime‚785à+dnÉ&ßÃM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21789474 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1563016592896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36904696 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21813672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14343922 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8823377 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 trainer/global_step‚849 _step‚66 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36880496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15091024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34205654044672 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8823375 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1595170468352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30906595076096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1558721529150 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5520547 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32642637451776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34205899497472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1563016592896 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30906612115456 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1558497887342 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚34205899497472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291418112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267478016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 328248320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 286640640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36645098708992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884173150071797e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21813672[m½3Í&àÇM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32642637451776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34205899497472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1563016592896 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30906612115456 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1558497887342 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚34205899497472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291418112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267478016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 328248320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 286640640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36645098708992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884173150071797e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21813672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21789474 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1563016592896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36904696 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21813672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14343922 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8823377 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 trainer/global_step‚849 _step‚66 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36880496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15091024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34205654044672 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8823375 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1595170468352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30906595076096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1558721529150 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5520547B˜Ñ»!0á+ _wandbruntime‚785  _runtime‚785ÂT«…âµ .loss/FBt1/detection_loss‚2.918899103214866 %loss/total_loss‚63.54364281981664 1loss/FBt2/assignment_loss‚0.28435703881012325 /loss/FRt3/detection_loss‚2.3995954543352127 0loss/SRqqt1/detection_loss‚2.975089654326439 1loss/SRqqt2/detection_loss‚1.3795030893959912 1loss/SRqqt4/detection_loss‚2.1941133379936217 trainer/global_step‚849 1loss/SRqqt2u2µ¶/assignment_loss‚2.377696877502533 2loss/SRqqt3/assignment_loss‚2.6351127726275747 .loss/FRt1/detection_loss‚2.974140200763941 0loss/FRt2/assignment_loss‚3.7223350922266643 /loss/FBt4/assignment_loss‚1.045759652342115 /loss/FRt4/detection_loss‚2.6252037033890234 0loss/SRqqt3/detection_loss‚2.527297980901672 .loss/FBt4/detection_loss‚3.294448971748352 2loss/SRqqt4/assignment_loss‚2.4746312816937763 epoch‚0 0loss/FBt1/assignment_loss‚0.6235772094484051 /loss/FBt2/detection_loss‚1.0385497136089903 -loss/FRt2/detection_loss‚2.55685801770952 .loss/FRt1/assignment_loss‚5.50653512775898 /loss/FRt3/assignment_loss‚4.364637730213312 1loss/SRqqt1/assignment_loss‚2.888602933713368 loss/FBt3/assignment_loss‚0 _runtime‚ 785.759462683 /loss/FRt4/assignment_loss‚4.412266933556759 ,loss/FBt3/detection_loss‚6.3244309425354 % _timestamp‚1.7884173150095708e+09 _step‚675óÔ?ã¹ 1loss/FBt2/assignment_loss‚0.28435703881012325 /loss/FRt3/detection_loss‚2.3995954543352127 0loss/SRqqt1/detection_loss‚2.975089654326439 1loss/SRqqt2/detection_loss‚1.3795030893959912 1loss/SRqqt4/detection_loss‚2.1941133379936217 trainer/global_step‚849 1loss/SRqqt2/assignment_loss‚2.377696877502533 2loss/SRqqt3/assignment_loss‚2.6351127726275747 .loss/FRt1/detection_loss‚2.974140200763941 0loss/FRt2/assignment_loss‚3.7223350922266643 /loss/FBt4/assignment_loss‚1.045759652342115 /loss/FRt4/detection_loss‚2.6252037033890234 0loss/SRqqt3/detection_loss‚2.527297980901672 .loss/FBt4/detection_loss‚3.294448971748352 2loss/SRqqt4/assignment_loss‚2.4746312816937763 epoch‚0 0loss/FBt1/assignment_loss‚0.6235772094484051 /loss/FBt2/detection_loss‚1.0385497136089903 -loss/FRt2/detection_loss‚2.55685801770952 .loss/FRt1/assignment_loss‚5.50653512775898 /loss/FRt3/assignment_loss‚4.364637730213312 1loss/SRqqt1/assignment_loss‚2.888602933713368 loss/FBt3/assignment_loss‚0 _runtime‚ 785.759462683 /loss/FRt4/assignment_loss‚4.412266933556759 ,loss/FBt3/detection_loss‚6.3244309425354 % _timestamp‚1.7884173150095708e+09 _step‚67 .loss/FBt1/detection_loss‚2.918899103214866 %loss/total_loss‚63.54364281981664C ¾dÅä‚(:‚ ¬ªäÔ‰µ¹x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘«f%å‚(:š ¬ªäÔÖÛÉz disk.dm-1.out‚ 28.94921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0) disk./.usagePercent‚73.66551439040879 disk.rbd2.out‚ 51.14453125 disk.rbd4.in‚0 disk.sda3.out‚ 28.94921875$ memory_percent‚97.84295558929443% disk./.usageGB‚1097.6693992614746 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚4 disk.dm-0.in‚4! disk.dm-0.out‚3075.5849609375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 28.94921875 network.sent‚ 1.11231e+06 network.recv‚353084$ disk.nvme0n1.out‚3075.5849609375 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4( proc.memory.availableMB‚ 353.41015625* proc.memory.percent‚16.820144653320312 disk.rbd1.in‚0 disk.drbd1777.out‚0 disk.sda3.in‚0 proc.memory.rssMB‚ 2755.8125 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd3.out‚0 cpu‚25.042393999161618& disk.nvme0c0n1.out‚3075.5849609375 disk.dm-1.in‚0 disk.rbd1.out‚ 0.36962890625u$úæ‚(:… »ªäÔƶ©x gpu.0.gpu‚20.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AyZ‹ç‚(:’ »ªäÔØî…{ cpu‚25.042501021214253 disk.sda3.out‚29.34375! disk.dm-0.out‚3174.6005859375 disk.dm-1.in‚0* proc.memory.percent‚16.820144653320312 disk.rbd3.in‚0$ disk.nvme0n1.out‚3174.6005859375 disk.rbd2.out‚ 51.14453125 disk.rbd0.out‚0 network.recv‚358713 proc.cpu.threads‚17) disk./.usagePercent‚73.66551439040879 disk.rbd1.out‚ 0.36962890625 disk.rbd0.in‚0 disk.rbd4.out‚0 network.sent‚ 1.161954e+06 disk.sda.in‚0 disk.dm-0.in‚4 disk.rbd4.in‚0 disk.sda1.in‚0$ memory_percent‚97.84295558929443 proc.memory.rssMB‚ 2755.8125 disk.sda3.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 353.41015625% disk./.usageGB‚1097.6693992614746 disk.rbd3.out‚0 disk.sda.out‚29.34375 disk.sda2.out‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚4 disk.nvme0n1.in‚4 disk.sda2.in‚0 disk.dm-1.out‚29.34375& disk.nvme0c0n1.out‚3174.6005859375 disk.sda1.out‚0>Ø›è‚(: ʪäÔ–±ðx gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.436' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.887200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fÊÌ)é‚(:ž ʪäÔÞ¨ƒ{ disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 353.3984375 network.sent‚ 1.167689e+06 disk.nvme0c0n1.in‚4' disk.nvme0c0n1.out‚3175.02587890625 disk.dm-0.in‚4" disk.dm-0.out‚3175.02587890625 disk.rbd2.in‚0 disk.sda3.out‚ 30.01171875$ memory_percent‚97.84302711486816 proc.cpu.threads‚17 network.recv‚362536 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd2.out‚ 52.828125 disk.rbd4.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚4 disk.sda2.out‚0 disk.dm-1.out‚ 30.01171875# proc.memory.rssMB‚ 2755.81640625* proc.memory.percent‚16.820168495178223 cpu‚25.042634453806752% disk.nvme0n1.out‚3175.02587890625$ disk./.usageGB‚1097.669319152832 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 30.01171875 disk.sda3.in‚0 disk.rbd1.out‚ 0.36962890625) disk./.usagePercent‚73.66550901425056h5¿0ê+ _wandbruntime‚830  _runtime‚830:þ¶ÈÄë¾ & lr-AdamW/pg2‚0.0005965959348887804 trainer/global_step‚899 % _timestamp‚1.7884173600146701e+09 _runtime‚ 830.764615338 _step‚68 & lr-AdamW/pg1‚0.0005965959348887804EYÄ™Èì & lr-AdamW/pg2‚0.0005965959348887804 trainer/global_step‚899 % _timestamp‚1.7884173600146701e+09 _runtime‚ 830.764615338 _step‚68 & lr-AdamW/pg1‚0.0005965959348887804Dr^+z0í+ _wandbruntime‚830  _runtime‚830̲™†¿'î¹O SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5838901 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36174274746368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15959255 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15959253 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1653305472000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5839560 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1653078035456 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9330772 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38753977197056 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9330770 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293528576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23044825 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15959253 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 trainer/global_step‚899 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36174520185856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚69 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚36174520185856 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1648299058950 $ _timestamp‚1.788417360015409e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39028252 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32685415511040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1653305472000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1687027368448 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚34521196710912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15170332"¾Ã'ï½O F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 830.765498943 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37067243357184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36174274746368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15959255 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15959253 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1653305472000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5839560 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1653078035456 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9330772 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38753977197056 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9330770 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293528576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23044825 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15959253 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 trainer/global_step‚899 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36174520185856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚69 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚36174520185856 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1648299058950 $ _timestamp‚1.788417360015409e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39028252 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32685415511040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1653305472000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1687027368448 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚34521196710912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15170332 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5838901 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15186771 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34372090108614 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚36214450384896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39047348 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37108389113344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39071548 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9341125 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15976972 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1688628841472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚36214695837696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 269575168 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23070376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚34371849427446 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38797017954816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚70 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23094574 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15187417 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5846290 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 293515264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1650131458550 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1655143179776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 328248320 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9341127 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39071548 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23094574 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚34559534654976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚32721735008256 % _timestamp‚1.7884173608798723e+09–ŽJÎ&òÈM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 831.630180591 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37108413053440 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1654915729920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15976972 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1688628841472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚36214695837696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 269575168 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23070376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚34371849427446 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38797017954816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚70 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23094574 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15187417 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5846290 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 293515264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1650131458550 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1655143179776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 328248320 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9341127 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39071548 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23094574 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚34559534654976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚32721735008256 % _timestamp‚1.7884173608798723e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23070376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38797311470080 trainer/global_step‚899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5845646 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1654915729920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1688898416640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚36214695837696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚34559552657920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚34559534654976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15186771 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34372090108614 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚36214450384896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39047348 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37108389113344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39071548 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9341125 Eö‚(:„ ÙªäÔ¡øÆx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.199' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8398 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øjuž(÷‚(: ÙªäÔÕÁ{# memory_percent‚97.8447437286377# proc.memory.rssMB‚ 2755.81640625) disk./.usagePercent‚73.66554306325263 disk.rbd1.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.820168495178223 network.recv‚367462% disk./.usageGB‚1097.6698265075684 disk.sda2.in‚0 disk.rbd2.out‚ 52.828125 disk.drbd1777.in‚0 disk.sda3.out‚ 30.52734375 network.sent‚ 1.173936e+06 disk.rbd3.in‚0 disk.nvme0n1.in‚4% disk.nvme0n1.out‚3274.11572265625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚4' disk.nvme0c0n1.out‚3274.11572265625 disk.sda.out‚ 30.52734375' proc.memory.availableMB‚ 353.1171875 disk.dm-0.in‚4 disk.dm-1.out‚ 30.52734375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda3.in‚0 cpu‚25.04273272942274 disk.sda1.in‚0 disk.rbd1.out‚ 0.36962890625" disk.dm-0.out‚3274.11572265625 disk.drbd1777.out‚0Qæ÷õø‚(:… èªäÔ’x gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.034' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù¼J‹&ù‚(:› èªäÔœÄ{% proc.memory.availableMB‚ 353.34375 network.recv‚372198 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda.out‚ 31.09765625# proc.memory.rssMB‚ 2755.81640625 network.sent‚ 1.222785e+06% disk./.usageGB‚1097.6698989868164 disk.rbd0.in‚0& disk.nvme0c0n1.out‚3373.3505859375 disk.sda1.out‚0 disk.dm-0.in‚4 disk.dm-1.out‚ 31.09765625 disk.rbd2.in‚0 disk.sda3.in‚0# memory_percent‚97.8433609008789 proc.cpu.threads‚17 disk.nvme0n1.in‚4 disk.sda2.in‚0 disk.nvme0c0n1.in‚4 disk.rbd3.in‚0 cpu‚25.042572096547605 disk.rbd1.out‚ 0.36962890625$ disk.nvme0n1.out‚3373.3505859375! disk.dm-0.out‚3373.3505859375 disk.rbd4.in‚0 disk.rbd2.out‚ 54.56640625* proc.memory.percent‚16.820168495178223) disk./.usagePercent‚73.66554792739576 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda3.out‚ 31.09765625†"ºiŽú‚(:ƒ ÷ªäÔÉÓx gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f¾öñ(û‚(: ÷ªäÔßÆÅ{ disk.dm-1.out‚ 31.55859375 disk.dm-0.in‚4 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda.in‚0$ disk.nvme0n1.out‚3472.4345703125! disk.dm-0.out‚3472.4345703125 cpu‚25.04269487648172 disk.drbd1777.in‚0 disk.nvme0c0n1.in‚4& disk.nvme0c0n1.out‚3472.4345703125 disk.rbd1.in‚0 disk.rbd1.out‚ 0.36962890625* proc.memory.percent‚16.820168495178223 disk.dm-1.in‚0 disk.drbd1777.out‚0' proc.memory.availableMB‚ 353.3203125 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2755.81640625 disk.sda1.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚4$ memory_percent‚97.84350395202637 network.sent‚ 1.228912e+06 network.recv‚377149 proc.cpu.threads‚17 disk.rbd2.out‚ 54.56640625 disk.rbd0.in‚0 disk.sda.out‚ 31.55859375 disk.sda3.out‚ 31.55859375 disk.sda2.out‚0 disk.rbd4.out‚0) disk./.usagePercent‚73.66554690336564% disk./.usageGB‚1097.6698837280273y|Il0ü+ _wandbruntime‚876  _runtime‚876k]>ÐÂý¼ # _timestamp‚1.78841740618714e+09 _runtime‚ 876.937068743 _step‚72 & lr-AdamW/pg1‚0.0006297770213675779 & lr-AdamW/pg2‚0.0006297770213675779 trainer/global_step‚949Q ÷ÆþÀ _step‚72 & lr-AdamW/pg1‚0.0006297770213675779 & lr-AdamW/pg2‚0.0006297770213675779 trainer/global_step‚949 # _timestamp‚1.78841740618714e+09 _runtime‚ 876.937068743HÖaÂ×0ÿ+ _wandbruntime‚876  _runtime‚876EP80À'€ºO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1740156271758 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24325727 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744977172480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40906137469440 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36240711714958 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293528576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34500538403840 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24349899 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40906430998016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1780702828032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269588480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36240471033998 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38183316526080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16845203 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6165894 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36438093914112 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚73 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38183071086592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36438093914112 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16845203 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38183316526080 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9848520 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39125728169984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1739932630158 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 % _timestamp‚1.7884174061878648e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1745204609024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24349899 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1745204609024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16014416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16013766 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41170930 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41195104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9848522 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34500555443200 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16845205 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36438111917056 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16845205 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6165246 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1744977172480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36438111917056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262˜ô«#,¾O TGDeviceStatsMonitor.on_train_batch_©¤O˜'start/allocation.large_pool.allocated‚16845205 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6165246 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1744977172480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36438111917056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1740156271758 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24325727 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744977172480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40906137469440 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36240711714958 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293528576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34500538403840 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24349899 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40906430998016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1780702828032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269588480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36240471033998 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38183316526080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16845203 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6165894 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36438093914112 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚73 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38183071086592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36438093914112 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16845203 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38183316526080 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9848520 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39125728169984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1739932630158 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 % _timestamp‚1.7884174061878648e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1745204609024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24349899 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1745204609024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16014416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16013766 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41170930 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41195104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9848522 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34500555443200 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16845205 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36438111917056IúbÐ0‚+ _wandbruntime‚877  _runtime‚877òÚžÊ&ƒÄM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6172209 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1747042316800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1746814866944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1782301188096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36476449861120 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1746814866944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚38223492177920 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40949470726656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34536840861696 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24375476 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16030456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _runtime‚ 877.829093106 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16031086 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 328248320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 286640640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38223492177920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36476449861120 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6171581 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9858875 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41238400 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24375476 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39166897866240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚38223246725120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 271672320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16862924 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24351278 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41214200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36476431858176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36476431858176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 % _timestamp‚1.7884174070783396e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9858877 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16862922 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 295612416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40949175114240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16862922 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41214200 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39166873926144 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36278605891454 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1782572860416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1741988671566 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16862924 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36278846572622 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚34536857901056 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _step‚74 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24351278 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41238400 trainer/global_step‚949p&Î&„ÈM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚38223246725120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 271672320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16862924 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24351278 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41214200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36476431858176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36476431858176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 % _timestamp‚1.7884174070783396e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9858877 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16862922 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 295612416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40949175114240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16862922 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41214200 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39166873926144 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36278605891454 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1782572860416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1741988671566 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16862924 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36278846572622 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚34536857901056 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _step‚74 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24351278 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41238400 trainer/global_step‚949 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6172209 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1747042316800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1746814866944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1782301188096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36476449861120 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1746814866944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚38223492177920 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40949470726656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34536840861696 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24375476 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16030456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _runtime‚ 877.829093106 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16031086 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 328248320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 286640640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38223492177920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36476449861120 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6171581 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9858875 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41238400 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24375476 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39166897866240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43361956 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1837103746048 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25606629 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16853364 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1831789842966 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1831566201366 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43337782 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚38355009120256 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43058764537344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25630801 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1837103746048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17731155 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1874286160384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38147468178966 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1836876309504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 trainer/global_step‚999 $ _timestamp‚1.788417452285806e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40191867426816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17731153 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25630801 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38354991117312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289334272 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40192112866304 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40191867426816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6487760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38354991117312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25606629 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16854032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10366270 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38147227498006 _step‚77’”œNÄ'“¾O H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40192112866304 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40191867426816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6487760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38354991117312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25606629 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16854032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10366270 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38147227498006 _step‚77 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36315661296640 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17731153 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43058475203072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 328248320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265394176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 227436544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10366272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41184189042688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1874551554560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚668 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43361956 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1837103746048 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25606629 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16853364 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1831789842966 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1831566201366 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43337782 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚38355009120256 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43058764537344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25630801 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1837103746048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17731155 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1874286160384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38147468178966 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1836876309504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 trainer/global_step‚999 $ _timestamp‚1.788417452285806e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40191867426816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17731153 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25630801 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38354991117312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289334272 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41225358738944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1876168528896 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17748872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25656378 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1838941453824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25632180 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43405252 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 _step‚78 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43405252 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 328248320 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43101822880256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17748874 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25632180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1838714003968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16871019 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1838714003968 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6494392 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 _runtime‚ 923.921696942 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25656378 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16870359 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41225382679040 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40232288518144 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36351980793856'éÆÓÐ&–ÊM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 _runtime‚ 923.921696942 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25656378 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16870359 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41225382679040 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40232288518144 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36351980793856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17748872 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17748874 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41225358738944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1876168528896 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17748872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25656378 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1838941453824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25632180 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43405252 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 _step‚78 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43405252 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 328248320 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43101822880256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17748874 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25632180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1838714003968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16871019 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1838714003968 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6494392NWïù0—+ _wandbruntime‚923  _runtime‚923Xuü†S˜Í 1loss/SRqqt1/assignment_loss‚2.236921031114667 0loss/FBt1/assignment_loss‚0.8593560189008713 0loss/FBt4/assignment_loss‚0.3927759053221962 % _timestamp‚1.7884174531729746e+09 0loss/FBt2/assignment_loss‚0.2942613675808389 .loss/FBt1/detection_loss‚2.791015738248825 /loss/FRt3/assignment_loss‚4.376087014739578 /loss/FRt4/assignment_loss‚4.550072355968196 1loss/SRqqt1/detection_loss‚2.2387409702289935 1loss/SRqqt3/detection_loss‚2.2422797890698036 /loss/FBt2/detection_loss‚0.8796662383479997 /loss/FBt4/detection_loss‚3.6423764526844025 1loss/SRqqt2/assignment_loss‚2.393140602338163 2loss/SRqqt3/assignment_loss‚2.2804250731700804 1loss/FBt3/assignment_loss‚0.48263349756597673 .loss/FRt1/detection_loss‚2.716961723727149 .loss/FRt3/detection_loss‚2.712748100628724 /loss/FRt2/detection_loss‚1.9532826873163383 /loss/FRt4/detection_loss‚2.3368258229116114 epoch‚0 /loss/FRt1/assignment_loss‚4.507182391914162 0loss/FRt2/assignment_loss‚3.3296325077613194 1loss/SRqqt4/assignment_loss‚2.310197084795597 0loss/SRqqt2/detection_loss‚1.349345016139972 .loss/FBt3/detection_loss‚4.019274145364761 _runtime‚ 923.922852089 _step‚79 1loss/SRqqt4/detection_loss‚2.2989755584750067 &loss/total_loss‚57.194177094315236 trainer/global_step‚999Ä]ø-W™Ñ &loss/total_loss‚57.194177094315236 trainer/global_step‚999 1loss/SRqqt1/assignment_loss‚2.236921031114667 0loss/FBt1/assignment_loss‚0.8593560189008713 0loss/FBt4/assignment_loss‚0.3927759053221962 % _timestamp‚1.7884174531729746e+09 0loss/FBt2/assignment_loss‚0.2942613675808389 .loss/FBt1/detection_loss‚2.791015738248825 /loss/FRt3/assignment_loss‚4.376087014739578 /loss/FRt4/assignment_loss‚4.550072355968196 1loss/SRqqt1/detection_loss‚2.2387409702289935 1loss/SRqqt3/detection_loss‚2.2422797890698036 /loss/FBt2/detection_loss‚0.8796662383479997 /loss/FBt4/detection_loss‚3.6423764526844025 1loss/SRqqt2/assignment_loss‚2.393140602338163 2loss/SRqqt3/assignment_loss‚2.2804250731700804 1loss/FBt3/assignment_loss‚0.48263349756597673 .loss/FRt1/detection_loss‚2.716961723727149 .loss/FRt3/detection_loss‚2.712748100628724 /loss/FRt2/detection_loss‚1.9532826873163383 /loss/FRt4/detection_loss‚2.3368258229116114 epoch‚0 /loss/FRt1/assignment_loss‚4.507182391914162 0loss/FRt2/assignment_loss‚3.3296325077613194 1loss/SRqqt4/assignment_loss‚2.310197084795597 0loss/SRqqt2/detection_loss‚1.349345016139972 .loss/FBt3/detection_loss‚4.019274145364761 _runtime‚ 923.922852089 _step‚79 1loss/SRqqt4/detection_loss‚2.2989755584750067O¤ˆY…›š‚(: ³«äÔÇ϶x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.259' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.251800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­°B$›‚(:™ ³«äÔøÞèz disk.rbd3.in‚0 disk.nvme0c0n1.in‚4 disk.sda1.in‚0 cpu‚25.043192982634075! disk.dm-0.out‚3770.7724609375 disk.sda2.out‚0( proc.memory.availableMB‚ 353.15234375$ disk./.usageGB‚1097.670150756836 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.36962890625 disk.dm-0.in‚4 disk.dm-1.out‚ 33.7265625 disk.rbd2.in‚0$ disk.nvme0n1.out‚3770.7724609375) disk./.usagePercent‚73.66556482389302& disk.nvme0c0n1.out‚3770.7724609375 disk.sda.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 network.sent‚ 1.295799e+06 disk.sda.out‚ 33.7265625 disk.sda3.out‚ 33.7265625 disk.nvme0n1.in‚4 disk.rbd0.in‚0# memory_percent‚97.8445291519165* proc.memory.percent‚16.820168495178223 disk.rbd3.out‚0 disk.rbd4.out‚0 disk.rbd2.out‚ 59.3828125# proc.memory.rssMB‚ 2755.81640625 proc.cpu.threads‚17 network.recv‚395415 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0†x›œ‚(: «äÔÆõÁx gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚125.979' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.195800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ“¡‚(:‘ «äÔŒÿÎz network.sent‚ 1.345273e+06 disk.dm-0.in‚4 disk.sda2.in‚0 disk.nvme0c0n1.in‚4$ memory_percent‚97.84610271453857 disk.rbd3.in‚0 disk.sda.in‚0) disk./.usagePercent‚73.66568386739645" disk.dm-0.out‚3870.79833984375 disk.sda2.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚59.625 proc.cpu.threads‚17% disk./.usageGB‚1097.6719245910645 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.36962890625 network.recv‚400445 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0* proc.memory.percent‚16.820168495178223' disk.nvme0c0n1.out‚3870.79833984375 disk.sda3.out‚34.3125# proc.memory.rssMB‚ 2755.81640625 cpu‚25.043308857670144 disk.sda1.out‚0 disk.sda.out‚34.3125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd4.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚4( proc.memory.availableMB‚ 352.89453125 disk.dm-1.out‚34.3125% disk.nvme0n1.out‚3870.79833984375 disk.rbd3.out‚0ÿÛ"›ž‚(: Ñ«äÔ ‘x gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜´»(Ÿ‚(: Ñ«äÔ³…Ç{ disk.rbd2.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 0.36962890625 cpu‚25.043108197752915 disk.dm-1.out‚ 34.8046875 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 34.8046875* proc.memory.percent‚16.820168495178223 disk.sda.in‚0 disk.sda.out‚ 34.8046875 disk.sda3.in‚0 network.sent‚ 1.351644e+06 disk.rbd2.out‚ 61.13671875' disk.nvme0c0n1.out‚3872.80224609375 disk.sda1.in‚0# memory_percent‚97.8482723236084' proc.memory.availableMB‚ 352.5390625 disk.dm-0.in‚4 disk.nvme0c0n1.in‚4 disk.nvme0n1.in‚4% disk.nvme0n1.out‚3872.80224609375 proc.cpu.threads‚17" disk.dm-0.out‚3872.80224609375 disk.drbd1777.in‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2755.81640625) disk./.usagePercent‚73.66565391451495% disk./.usageGB‚1097.6714782714844 disk.dm-1.in‚0 disk.rbd4.in‚0 network.recv‚405501ëN-¤0 + _wandbruntime‚968  _runtime‚968ã×´{Å¡¿ % _timestamp‚1.7884174977427018e+09 _runtime‚ 968.492639955 _step‚80 & lr-AdamW/pg1‚0.0006961391943251732 & lr-AdamW/pg2‚0.0006961391943251732 trainer/global_step‚1049Kê ɢà & lr-AdamW/pg1‚0.0006961391943251732 & lr-AdamW/pg2‚0.0006961391943251732 trainer/global_step‚1049 % _timestamp‚1.7884174977427018e+09 _runtime‚ 968.492639955 _step‚80PDS¦0£+ _wandbruntime‚968  _runtime‚968é3ˆbÂ'¤¼O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1929002883072 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45210663514112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45504634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10884022 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚ 968.493564668 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40271906323456 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38130784189440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 trainer/global_step‚1049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6809853 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 $ _timestamp‚1.788417497743444e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40271906323456 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42200909206528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17694539 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40271888320512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1928775446528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40054224642974 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26887531 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18617103 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 328248320 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _step‚81 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18617103 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271685632 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42200663767040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40053983962014 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38130801228800 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541125632 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6810517 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1923199772574 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45528808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1923423414174 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26911703 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26911703 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1928775446528 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42200663767040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40271888320512 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45210959139840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 295625728 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18617105 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1968261344256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43242673855488 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1967989658624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10884020 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42200663767040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40053983962014 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38130801228800 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/alloca3HAted_bytes.small_pool.peak‚ 541125632 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6810517 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1923199772574 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45528808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1923423414174 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26911703 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26911703 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1928775446528 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42200663767040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40271888320512 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45210959139840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 295625728 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18617105 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1968261344256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43242673855488 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1967989658624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10884020 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10884022 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚ 968.493564668 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40271906323456 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38130784189440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 trainer/global_step‚1049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6809853 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 $ _timestamp‚1.788417497743444e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40271906323456 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42200909206528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17694539 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40271888320512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1928775446528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40054224642974 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561Q:ï¨0¦+ _wandbruntime‚969  _runtime‚969¡É¡ÝÍ&§ÇM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26913082 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10894375 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40092359500638 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26937280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17710468 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1925032172174 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42240839405568 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 293515264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45547904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10894377 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43283867491840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 286640640 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40092118819470 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45547904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 328248320 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38167086647296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26913082 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45253706873344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40310244267520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 269575168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40310244267520 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45572104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6816093 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1930613140992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18634822 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1930613140992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45572104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1970132896768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6816742 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38167103686656¹;ƒ×Ñ&¨ËM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1970132896768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6816742 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38167103686656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26913082 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10894375 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40092359500638 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26937280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17710468 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1925032172174 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42240839405568 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 293515264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45547904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10894377 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43283867491840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 286640640 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40092118819470 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45547904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 328248320 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38167086647296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26913082 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45253706873344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40310244267520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 269575168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40310244267520 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45572104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6816093 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1930613140992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18634822 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1930613140992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45572104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47695660 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28168433 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45301182608384 trainer/global_step‚1099 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291431424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267491328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19503055 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7133311 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18534406 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44209460107264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11401772 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262ÁLþ,Ç'·ÁO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39945924121600 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41960981106982 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚39945907082240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 _runtime‚ 1013.66475057 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42188785523712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41960740426022 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2020902020096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 328248320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚677 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47363218061312 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚47362926629888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42188785523712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2020674583552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7132636 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2062035452928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2061767961600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884175429145532e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44209705546752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42188803526656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18535083 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44209460107264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11401772 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19503055 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19503053 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45301158668288 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47671486 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44209460107264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2020902020096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 _step‚85 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47695660 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28168433 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45301182608384 trainer/global_step‚1099 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291431424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267491328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6738577920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19503055 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7133311 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 245439488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18534406 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28193984 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2022739727872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28193984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18551050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28218182 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261186560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39982209540096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28218182 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 _step‚86 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11412125 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42227123467776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2022739727872 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 _runtime‚1014.512995111 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11412127 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2022512278016 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19520772 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19520774 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19520774 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44249881198592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42227141470720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47405972031488 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44249635745792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 286640640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19520772 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45342328364544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42227141470720 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47406257158144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44249881198592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2016665743382 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 % _timestamp‚1.7884175437624135e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2022512278016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2016889385190 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47738956 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18551689 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47738956 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7139562 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7138925 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285126656 UHDeviceStatsM%Æñ*onitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47714756 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚41998875283478 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47714756 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45342352304640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561¡ÂÆÒ&ºÌM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2022739727872 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 _runtime‚1014.512995111 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11412127 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2022512278016 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19520772 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19520774 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19520774 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚29899 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44249881198592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42227141470720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47405972031488 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44249635745792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 286640640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19520772 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45342328364544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42227141470720 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47406257158144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44249881198592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2016665743382 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 % _timestamp‚1.7884175437624135e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2022512278016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2016889385190 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47738956 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18551689 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47738956 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7139562 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7138925 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285126656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47714756 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚41998875283478 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47714756 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45342352304640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223641808 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28193984 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2022739727872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28193984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18551050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28218182 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261186560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39982209540096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28218182 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 _step‚86 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11412125 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42227123467776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0VºTÖü2»- _wandbruntime‚1014  _runtime‚1014j“Ï-P¼Ê 0loss/FBt3/assignment_loss‚0.7643551230430603 1loss/SRqqt3/detection_loss‚1.9472415283614515 .loss/FBt1/detection_loss‚5.431926588217418 .loss/FBt4/detection_loss‚3.610654020309448 #loss/total_loss‚58.965657443673 _runtime‚1014.513980077 .loss/FRt3/detection_loss‚2.353944167177728 /loss/FBt2/detection_loss‚0.8700040868307287 epoch‚0 0loss/FBt4/assignment_loss‚0.7662984967231751 .loss/FRt3/assignment_loss‚4.17128106127394 2loss/SRqqt1/assignment_loss‚2.4820067066895333 2loss/SRqqt2/assignment_loss‚1.9046585629247639 _step‚87 0loss/SRqqt2/detection_loss‚1.350441281517891 % _timestamp‚1.7884175437641315e+09 /loss/FRt1/detection_loss‚2.5634212410727213 1loss/FBt2/assignment_loss‚0.19573111928662615 /loss/FRt4/assignment_loss‚4.904628102843826 .loss/FBt3/detection_loss‚3.252901941537857 0loss/FBt1/assignment_loss‚0.4443792724632658 /loss/FRt4/detection_loss‚2.5164410439697473 1loss/SRqqt1/detection_loss‚2.5452599580350674 trainer/global_step‚1099 /loss/FRt2/detection_loss‚2.1623648485789695 0loss/FRt2/assignment_loss‚2.7626049034297466 /loss/SRqqt3/assignment_loss‚2.3687004236614 2loss/SRqqt4/assignment_loss‚2.8544180032818818 /loss/FRt1/assignment_loss‚4.650625969088355 0loss/SRqqt4/detection_loss‚2.091368993354398 SmèT½Î 0loss/SRqqt2/detection_loss‚1.350441281517891 % _timestamp‚1.7884175437641315e+09 /loss/FRt1/detection_loss‚2.5634212410727213 1loss/FBt2/assignment_loss‚0.19573111928662615 /loss/FRt4/assignment_loss‚4.904628102843826 .loss/FBt3/detection_loss‚3.252901941537857 0loss/FBt1/assignment_loss‚0.4443792724632658 /loss/FRt4/detection_loss‚2.5164410439697473 1loss/SRqqt1/detection_loss‚2.5452599580350674 trainer/global_step‚1099 /loss/FRt2/detection_loss‚2.1623648485789695 0loss/FRt2/assignment_loss‚2.7626049034297466 /loss/SRqqt3/assignment_loss‚2.3687004236614 2loss/SRqqt4/assignment_loss‚2.8544180032818818 /loss/FRt1/assignment_loss‚4.650625969088355 0loss/SRqqt4/detection_loss‚2.091368993354398 0loss/FBt3/assignment_loss‚0.7643551230430603 1loss/SRqqt3/detection_loss‚1.9472415283614515 .loss/FBt1/detection_loss‚5.431926588217418 .loss/FBt4/detection_loss‚3.610654020309448 #loss/total_loss‚58.965657443673 _runtime‚1014.513980077 .loss/FRt3/detection_loss‚2.353944167177728 /loss/FBt2/detection_loss‚0.8700040868307287 epoch‚0 0loss/FBt4/assignment_loss‚0.7662984967231751 .loss/FRt3/assignment_loss‚4.17128106127394 2loss/SRqqt1/assignment_loss‚2.4820067066895333 2loss/SRqqt2/assignment_loss‚1.9046585629247639 _step‚87W郾‚(:„ ¬äÔªèÃx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.434' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ4áÕ¿‚(:” ¬äÔêõÕ{ network.sent‚ 1.418469e+06 network.recv‚423553 disk.rbd0.out‚0 disk.rbd1.out‚ 0.36962890625# disk.nvme0n1.out‚4170.802734375* proc.memory.percent‚16.820240020751953 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.dm-0.in‚4 disk.dm-0.out‚4170.802734375 disk.sda2.out‚0% disk.nvme0c0n1.out‚4170.802734375 disk.sda3.in‚0! proc.memory.rssMB‚ 2755.828125% disk./.usageGB‚1097.6717414855957 disk.nvme0c0n1.in‚4 disk.rbd3.out‚0 disk.sda.out‚ 36.6640625 disk.rbd1.in‚0 disk.nvme0n1.in‚4 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0$ memory_percent‚97.84884452819824' proc.memory.availableMB‚ 352.4453125) disk./.usagePercent‚73.66567157903481 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda3.out‚ 36.6640625 cpu‚25.04349709965447 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 36.6640625 disk.rbd2.out‚ 64.4921875 disk.sda1.in‚0 disk.rbd3.in‚0"£¸ËÀ‚(:… œ¬äÔÕ°œx gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=á$Á‚(:™ œ¬äÔ¾ÇÁz disk.sda3.out‚ 37.29296875 disk.rbd0.in‚0% disk.nvme0c0n1.out‚4270.537109375* proc.memory.percent‚16.820263862609863 cpu‚25.04364582507443 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚4$ memory_percent‚97.84882068634033# proc.memory.rssMB‚ 2755.83203125# disk.nvme0n1.out‚4270.537109375 disk.dm-0.out‚4270.537109375 disk.sda2.out‚0 disk.sda1.out‚0 network.sent‚ 1.467546e+06 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚4 disk.dm-1.in‚0 disk.rbd4.out‚0 disk.sda1.in‚0% disk./.usageGB‚1097.6720695495605 disk.rbd1.out‚ 0.36962890625 disk.rbd2.in‚0 disk.rbd2.out‚ 66.296875 disk.sda.out‚ 37.29296875 disk.sda3.in‚0 disk.nvme0n1.in‚4( proc.memory.availableMB‚ 352.44921875 network.recv‚428068) disk./.usagePercent‚73.66569359568275 disk.dm-1.out‚ 37.29296875 disk.rbd0.out‚0 disk.rbd3.in‚0ºWO‚(:„ «¬äÔìê†x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚92.251' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ÂÂ(:” «¬äÔÏ€ª{ network.recv‚432557 disk.sda.out‚ 37.734375 disk.rbd1.in‚0 disk.rbd1.out‚ 0.36962890625% disk./.usageGB‚1097.6720695495605 disk.sda.in‚0 disk.dm-0.in‚4 disk.dm-1.out‚ 37.734375) disk./.usagePercent‚73.66569359568275 disk.rbd0.out‚0 disk.rbd4.in‚0$ disk.nvme0n1.out‚4369.6005859375$ memory_percent‚97.84903526306152* proc.memory.percent‚16.820263862609863 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.sda2.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 66.296875 proc.cpu.threads‚17 disk.drbd1777.in‚0& disk.nvme0c0n1.out‚4369.6005859375 cpu‚25.0434631914307 network.sent‚ 1.473622e+06 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 352.4140625# proc.memory.rssMB‚ 2755.83203125 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 37.734375! disk.dm-0.out‚4369.6005859375 disk.sda2.in‚0 disk.nvme0c0n1.in‚4 disk.sda1.in‚0 disk.nvme0n1.in‚4 disk.dm-1.in‚0ÑQ“–2Ä- _wandbruntime‚1058  _runtime‚1058ì`ñÆÅÀ & lr-AdamW/pg1‚0.0007625013672827683 & lr-AdamW/pg2‚0.0007625013672827683 trainer/global_step‚1149 % _timestamp‚1.7884175878551695e+09 _runtime‚1058.605127499 _step‚88&· oÊÆÄ _step‚88 & lr-AdamW/pg1‚0.0007625013672827683 & lr-AdamW/pg2‚0.0007625013672827683 trainer/global_step‚1149 % _timestamp‚1.7884175878551695e+09 _runtime‚1058.605127499X¥& Ü2Ç- _wandbruntime‚1058  _runtime‚1058Ñ@ø‘Ä'ȾO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19376321 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20389005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46218256447488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19375665 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47359643481088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46218501886976 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41761029975040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44105700729856 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41761047014400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44105682726912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49838338 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49862512 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43867737570990 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2106466914990 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49515468333056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2155535517696 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7456145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29449335 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289334272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2106690556590 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44105682726912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11919520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46218256447488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11919522 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265394176 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 227436544 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46218256447488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19375665 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitorªŠªÙ.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47359643481088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46218501886976 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚29899 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41761029975040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44105700729856 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41761047014400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44105682726912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49838338 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49862512 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43867737570990 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2106466914990 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49515468333056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2155535517696 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 245439488 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7456145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29449335 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289334272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2106690556590 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44105682726912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11919520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46218256447488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11919522 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265394176 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44144020670976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚90 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 $ _timestamp‚1.788417588703852e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49881608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20406722 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 250700800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20406724 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29474886 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49558265947136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2108299314590 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20406722 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2157703470592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚43905872428654 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1149 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 274640896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚46258677538816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19393070 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43905631747486 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19392452 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11929877 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49905808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11929875 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49881608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46258677538816 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44144038673920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2114411415040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47400837117440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚46258677538816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19393070 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43905631747486 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19392452 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11929877 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49905808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11929875 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49881608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46258677538816 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44144038673920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2114411415040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47400837117440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24198 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44144020670976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚90 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 $ _timestamp‚1.788417588703852e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6738577920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541125632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49881608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 227449856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20406722 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 250700800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541125632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20406724 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29474886 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49558265947136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2108299314590 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚29670 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20406722 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2157703470592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚43905872428654 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1149 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 274640896Z Lª2Í- _wandbruntime‚1059  _runtime‚1059“iTÎÎ /loss/FBt4/detection_loss‚3.1522783090670905 1loss/SRqqt4/assignment_loss‚2.512805775954173 .loss/FBt1/detection_loss‚2.136420543704714 %loss/total_loss‚55.75085680071727 _runtime‚1059.455597838 0loss/FRt4/assignment_loss‚4.7837469154788606 2loss/SRqqt3/assignment_loss‚1.9069056039055188 0loss/FBt2/assignment_loss‚0.1948474512451518 .loss/FRt3/detection_loss‚2.468070373535156 .loss/FRt4/detection_loss‚2.744872969004416 1loss/SRqqt2/detection_loss‚1.2715584004504814 /loss/FBt2/detection_loss‚0.9367724501413025 2loss/SRqqt1/assignment_loss‚2.4037107874949775 1loss/SRqqt2/assignment_loss‚2.294020997463829 1loss/FBt4/assignment_loss‚0.33942176800898477 0loss/SRqqt1/detection_loss‚2.453580497039689 /loss/FRt3/assignment_loss‚4.411884783506394 epoch‚0 $ _timestamp‚1.788417588705791e+09 /loss/FBt1/assignment_loss‚0.875070891210011 0loss/SRqqt4/detection_loss‚2.407282394476426 trainer/global_step‚1149 _step‚91 1loss/SRqqt3/detection_loss‚1.8166414878020685 .loss/FRt1/detection_loss‚2.745379631335919 /loss/FRt2/detection_loss‚1.8955744187983254 /loss/FRt1/assignment_loss‚4.227312540396666 0loss/FRt2/assignment_loss‚3.1067833995062206 1loss/FBt3/assignment_loss‚0.04657303774729371 /loss/FBt3/detection_loss‚4.6193413734436035wê¦zXÏÒ /loss/FBt3/detection_loss‚4.6193413734436035 /loss/FBt4/detection_loss‚3.1522783090670905 1loss/SRqqt4/assignment_loss‚2.512805775954173 .loss/FBt1/detection_loss‚2.136420543704714 %loss/total_loss‚55.75085680071727 _runtime‚1059.455597838 0loss/FRt4/assignment_loss‚4.7837469154788606 2loss/SRqqt3/assignment_loss‚1.9069056039055188 0loss/FBt2/assignment_loss‚0.1948474512451518 .loss/FRt3/detection_loss‚2.468070373535156 .loss/FRt4/detection_loss‚2.744872969004416 1loss/SRqqt2/detection_loss‚1.2715584004504814 /loss/FBt2/detection_loss‚0.9367724501413025 2loss/SRqqt1/assignment_loss‚2.4037107874949775 1loss/SRqqt2/assignment_loss‚2.294020997463829 1loss/FBt4/assignment_loss‚0.33942176800898477 0loss/SRqqt1/detection_loss‚2.453580497039689 /loss/FRt3/assignment_loss‚4.411884783506394 epoch‚0 $ _timestamp‚1.788417588705791e+09 /loss/FBt1/assignment_loss‚0.875070891210011 0loss/SRqqt4/detection_loss‚2.407282394476426 trainer/global_step‚1149 _step‚91 1loss/SRqqt3/detection_loss‚1.8166414878020685 .loss/FRt1/detection_loss‚2.745379631335919 /loss/FRt2/detection_loss‚1.8955744187983254 /loss/FRt1/assignment_loss‚4.227312540396666 0loss/FRt2/assignment_loss‚3.1067833995062206 1loss/FBt3/assignment_loss‚0.04657303774729371[)ýÊЂ(:„ º¬äÔóŠ„x gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.181' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?íBŒ"Ñ‚(:— º¬äÔܶ«z( disk./.usagePercent‚73.6656871954944 disk.sda2.in‚0 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.nvme0n1.in‚4* proc.memory.percent‚16.820287704467773$ disk./.usageGB‚1097.671974182129 disk.rbd2.in‚0 disk.sda.out‚ 38.2734375 disk.dm-0.in‚4 disk.drbd1777.in‚0& disk.nvme0c0n1.out‚4469.0576171875" proc.memory.rssMB‚ 2755.8359375 disk.sda3.out‚ 38.2734375$ disk.nvme0n1.out‚4469.0576171875 cpu‚25.043599741332837 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 38.2734375 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 352.15234375 proc.cpu.threads‚17 network.sent‚ 1.479366e+06 network.recv‚436389! disk.dm-0.out‚4469.0576171875 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 68.0390625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4 disk.rbd3.in‚0 disk.rbd1.out‚ 0.36962890625# memory_percent‚97.8506326675415yÒbïÒ‚(:… ɬäÔ‰ö²x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.011' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ¾Ò•$Ó‚(:™ ɬäÔ†ÉÃz disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚4568.1279296875 disk.rbd3.in‚0" proc.memory.rssMB‚ 2755.8359375* proc.memory.percent‚16.820287704467773 cpu‚25.043677504991276) disk./.usagePercent‚73.64823746595304 disk.rbd0.in‚0 disk.rbd4.in‚0$ disk.nvme0n1.out‚4568.1279296875 network.recv‚441752 disk.rbd2.out‚ 68.0390625 disk.rbd3.out‚0 disk.sda.out‚ 38.9296875 disk.nvme0n1.in‚4 network.sent‚ 1.528321e+06 disk.rbd2.in‚0 disk.drbd1777.out‚0! disk.dm-0.out‚4568.1279296875$ memory_percent‚97.84789085388184 proc.cpu.threads‚17% disk./.usageGB‚1097.4119606018066 disk.nvme0c0n1.in‚4 disk.sda1.out‚0 disk.sda3.out‚ 38.9296875 disk.dm-1.in‚0 disk.rbd4.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.36962890625 disk.rbd0.out‚0 disk.dm-0.in‚4' proc.memory.availableMB‚ 352.6015625 disk.dm-1.out‚ 38.9296875 disk.drbd1777.in‚0 disk.sda1.in‚0Jž™Ô‚(:Ž جäÔ³·Ðx gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚93.37' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.674000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hXu Õ‚(:— جäÔàÅïz disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.820287704467773 network.sent‚ 1.533515e+06 disk.rbd4.out‚0$ disk.nvme0n1.out‚4735.5947265625% disk./.usageGB‚1096.6101150512695 disk.sda1.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 39.45703125 cpu‚25.043523178031645& disk.nvme0c0n1.out‚4735.5947265625 disk.sda1.out‚0 disk.sda.out‚ 39.45703125# memory_percent‚97.8494644165039 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda.in‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 2755.8359375 network.recv‚445316) disk./.usagePercent‚73.59442493826327 disk.rbd1.out‚ 0.36962890625% proc.memory.availableMB‚ 352.34375 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd 9/_1777.in‚0 disk.drbd1777.out‚0 disk.sda3.out‚ 39.45703125 disk.dm-0.in‚4! disk.dm-0.out‚4735.5947265625 disk.rbd2.out‚69.78125 disk.rbd4.in‚0 disk.nvme0c0n1.in‚4 disk.rbd3.in‚0 disk.nvme0n1.in‚4 disk.dm-1.in‚0¬êóˆ2Ö- _wandbruntime‚1103  _runtime‚1103»goHÆ×À & lr-AdamW/pg1‚0.0007956824537615659 & lr-AdamW/pg2‚0.0007956824537615659 trainer/global_step‚1199 % _timestamp‚1.7884176328684447e+09 _runtime‚1103.618368252 _step‚92N¡¥ÊØÄ % _timestamp‚1.7884176328684447e+09 _runtime‚1103.618368252 _step‚92 & lr-AdamW/pg1‚0.0007956824537615659 & lr-AdamW/pg2‚0.0007956824537615659 trainer/global_step‚1199\IF~32Ù- _wandbruntime‚1103  _runtime‚1103ížÄRÄ'Ú¾O KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537287125 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12437270 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6738577920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚46022579930112 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45774253354038 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51667441571328 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _runtime‚1103.619267296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚1199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21274955 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2198324127798 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12437272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48227052787712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚46022597933056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2204700294144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2204472857600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49418152233984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚46022579930112 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43576152867840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51667735099904 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48227298227200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2204472857600 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7779664 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30754409 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2249313277440 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2198100486198 % _timestamp‚1.7884176328691585e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚29670 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24174 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚29670 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45774253354038 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51667441571328 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _runtime‚1103.619267296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240680960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚1199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21274955 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24172 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 286640640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2198324127798 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12437272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48227052787712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚46022597933056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2204700294144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2204472857600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚29899 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49418152233984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚46022579930112 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541125632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522520525 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43576152867840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24172 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51667735099904 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24174 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223641600]åìõ“2Ü- _wandbruntime‚1104  _runtime‚1104`ŽèÎ&ÝÈM EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30755788 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 % _timestamp‚1.7884176337476728e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20233667 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12447627 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240681168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚46060917874176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2206538001920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚1199 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51710822810112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2200156527606 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚46060935877120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259089408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21292674 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48267473879040 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48267228426240 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52072660 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283029504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49459297990144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚45812628892662 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21292672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7786699 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51710539780608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21292674 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20234326 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2251500879872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30779986 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21292672 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30755788 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 % _timestamp‚1.7884176337476728e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20233667 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12447627 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240‹C (š 681168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚46060917874176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2206538001920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚1199 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51710822810112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2200156527606 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537287125 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 245452800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚46060935877120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 245452800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 227449856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6738577920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24198 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259089408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21292674 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48267473879040 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48267228426240 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52072660 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283029504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49459297990144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚45812628892662 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚29899 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21292672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7786699 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51710539780608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21292674 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20234326 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2251500879872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522520525 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30779986 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21292672 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JPúžpbk HDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed"trainer/global_step2:JPÏè4Hgbb ?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections"trainer/global_step2:JPœdÜí]‘bX 5DeviceStatsMonitor.on_validation_batch_start/num_ooms"trainer/global_step2:JP=5x’bs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP²;_w“br ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP*ò@q”bl IDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current"trainer/global_step2:JPlŽTào•bj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JPdâ±±v–bq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPŽ&Òt—bo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPm „ûr˜bm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP–µ˜(w™br ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPò¤[…lšbg DDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed"trainer/global_step2:JP(t›bo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP¿Ísxœbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPDBrbm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPœ’¥8jžbe BDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated"trainer/global_step2:JP{>à@rŸbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPâÿÅ{c b^ ;DeviceStatsMonitor.on_validation_batch_start/max_split_size"trainer/global_step2:JPˆ9šûf¡ba >DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP×TDu¢bp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP½;eòn£bi FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed"trainer/global_step2:JPƦ$d¤b_ o°bj GDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak"trainer/global_step2:JPõË0s±bn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP²Ðl‘q²bl IDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated"trainer/global_step2:JPÑ&1×z³bu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPÛ ¹i´bd ADeviceStatsMonitor.on_validation_batch_start/active.all.allocated"trainer/global_step2:JPK@gµbb ?DeviceStatsMonitor.on_validation_batch_start/active.all.current"trainer/global_step2:JPðÖ s¶bn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JP¥×ô[u·bp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP¦à´,q¸bl IDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated"trainer/global_step2:JPåoLjw¹br ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP*ÑÊoºbj GDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak"trainer/global_step2:JPCþ4ts»bn KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPþÜr¼bm JDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPæOxe½b` =DeviceStatsMonitor.on_validation_batch_start/num_device_alloc"trainer/global_step2:JP×ÕÔ²i¾bd ADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams"trainer/global_step2:JPk{?'x¿bs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPÔk"ÓtÀbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPXŠ˜uÁbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPêIÀ yÂbt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPNßç±oÃbj GDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current"trainer/global_step2:JPÇ9½lÄbg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JPfU(ìoÅbj GDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current"trainer/global_step2:JP™ÒÖmÆbh EDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed"trainer/global_step2:JPⲤÓ|Çbw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPqÔ½ÔnÈbi FDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak"trainer/global_step2:JP¥Ï¥üwÉbr ODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP~Êby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPš24¼zËbu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPYi²tÌbo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPx/qyÍbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPEIÈoÎbj GDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current"trainer/global_step2:JP.Œ9ºtÏbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPñãÊ>iÐbd ADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed"trainer/global_step2:JPÍþðMeÑb` =DeviceStatsMonitor.on_validation_batch_start/active.all.freed"trainer/global_step2:JP¯CãqoÒbj GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated"trainer/global_step2:JPpª^jÓbe BDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak"trainer/global_step2:JP·Ü°yÔbt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP”IôIvÕbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPÏyÉtÖbo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP.Ìãp×bk HDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed"trainer/global_step2:JP㼿ÄlØbg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak"trainer/global_step2:JPù€ðtÙbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP°H45pÚbk HDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current"trainer/global_step2:JP;yüvÛbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPÇa«¢tÜbo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPò§ïnÝbi FDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current"trainer/global_step2:JP-æñ‚sÞbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP>çoßbj GDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed"trainer/global_step2:JPA_ënàbi FDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current"trainer/global_step2:JP>Dz„lábg DDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed"trainer/global_step2:JP›®é mâbh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JPz™ rãbm JDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current"trainer/global_step2:JP_gS+väbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP [N)2å- _wandbruntime‚1164  _runtime‚1164 DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884176941031454e+09 _runtime‚1164.853477647 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2242167864320 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚47387735947766 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚31396007 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 _step‚96‰s7./*ç©T J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚29899 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚51173183798272 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚561 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚299 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚31415422 ]JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚49929215244800 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6738577920 ^QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚90898432 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚19415 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚53599431 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚49929025933312 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 189311488 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6522520525 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 167818124 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚12980056 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚29670 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 189311488 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19417 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚22184009 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 6893338624 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚47686876071936 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 541125632 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚21091954 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚45152071959552 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚262 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚31396007 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚49929025933312 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚12980058 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1243 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚31396007 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 _step‚96 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚386 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚47686876071936 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2242167864320 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 171308544 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚22184007 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 171308544 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚8112282 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚47387920805250 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚561 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚47686858068992 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884176941031454e+09 _runtime‚1164.853477647 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2242167864320 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚47387735947766 B±y7qðbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPYü§õoñbj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JP<œEËiòbd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JP_m¾Êoóbj GDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated"trainer/global_step2:JPþ1pôbk HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPßêãåuõbp MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPÀçkïjöbe BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JP}k#¹n÷bi FDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated"trainer/global_step2:JP£„šjløbg DDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current"trainer/global_step2:JPɾtlùbg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JP²¸ç$júbe BDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak"trainer/global_step2:JP»;qûbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPÝ)abüb] :DeviceStatsMonitor.on_validation_batch_end/num_device_free"trainer/global_step2:JPBªUlýbg DDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed"trainer/global_step2:JPG‡Ð´rþbm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP‡â fÿba >DeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JPŸõNõd€b_ B÷tªbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPj¹Èóp«bk HDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JP;sÿIz¬bu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP­;5¸u­bp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPãçë,k®bf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JPâ4oÃk¯bf CDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP1íÛóc°b^ ;DeviceStatsMonitor.on_validation_batch_end/active.all.freed"trainer/global_step2:JPé±ðu±bp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPÖ¥ìºr²bm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPL)“?p³bk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JPŽ¸òón´bi FDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed"trainer/global_step2:JP<¨|µbw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPI}Š[¶bV 3DeviceStatsMonitor.on_validation_batch_end/num_ooms"trainer/global_step2:JP€wºb·b] :DeviceStatsMonitor.on_validation_batch_end/active.all.peak"trainer/global_step2:JP½™o¸bj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JP]lÈw¹br ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPBáíxºbs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP=jäj»be BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPPä4q¼bl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPQ–Õk½bf CDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed"trainer/global_step2:JPáÿ[÷i¾bd ADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak"trainer/global_step2:JP~©î¯w¿br ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPít> sÀbn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP8ø— gÁbb ?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated"trainer/global_step2:JP5jgÂbb ?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed"trainer/global_step2:JPâ0‹zÃbu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP?gÖ eÄb` =DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections"trainer/global_step2:JPéeEÔrÅbm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP&o…oÆbj GDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JP>ÉcÇb^ ;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak"trainer/global_step2:JPÆ_ètÈbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPßúWiÉbd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JPEÁ4âpÊbk HDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current"trainer/global_step2:JPÀ8³ÐxËbs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP_5PêhÌbc @DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak"trainer/global_step2:JPE§Š÷kÍbf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP×­lÝXÎbh EDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current"trainer/glØ]¥Pobal_step2:JP˜LͬrÏbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP"í§tÐbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPÌçanÑbi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JP°¼8¼oÒbj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPS ùfÓba >DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JP~dîrÔbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPòg[sÕbn KDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP¡ÈÿœnÖbi FDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current"trainer/global_step2:JPÈ[Óm×bh EDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current"trainer/global_step2:JPzDJhØbc @DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated"trainer/global_step2:JPœŽpœpÙbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP±±°ÃrÚbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPñŒåfkÛbf CDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated"trainer/global_step2:JP$g§cÜb^ ;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc"trainer/global_step2:JP­œ£ÄjÝbe BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JPQ¹yjÞbe BDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed"trainer/global_step2:JPàªÒÝqßbl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP¸­×pàbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPy€—|rábm JDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPó¢ÿ=qâbl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JPîR­“2ã- _wandbruntime‚1165  _runtime‚1165ruØ4)ä®R XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 537287125 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚29899 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚12989020 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 171398656 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 184857484 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚561 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚47712998858752 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 541125632 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚53622643 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚22198943 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2242560659456 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚22198945 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2236055165110 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚387 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚45176841948160 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 189311488 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚23 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚262 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚21106879 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚47413064931394 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚53488690232832 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚31404283 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚22198945 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19415 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1261 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚53488518834176 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚47412880073910n_Q8)å²R SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚31423698 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 541125632 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚29899 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚12989020 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 171398656 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 184857484 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚561 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚47712998858752 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 541125632 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚53622643 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚22198943 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2242560659456 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚22198945 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2236055165110 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚387 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚45176841948160 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 189311488 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚23 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚262 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚21106879 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚47413064931394 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚53488690232832 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚31404283 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚22198945 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19415 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1261 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚53488518834176 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚47412880073910 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 537287125 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 BNèbI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPçÊáJébE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP"¾®ÛJêbE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP· *JëbE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP°Ù«?NìbI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP†¯ÉJíbE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP_ ®ûHîbC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPròßNïbI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPf 6JðbE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPIWJñbE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPßséÝ;òb6 jet/accuracy_5_of_5"trainer/global_step2:JP¸ƒŽ•HóbC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPKÙ4;ôb6 jet/accuracy_1_of_6"trainer/global_step2:JPÁ/n'HõbC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP¦G_Æ;öb6 loss/val_total_loss"trainer/global_step2:JP¸,c4N÷bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPñ™tHøbC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP\W½JùbE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP&’ÛyHúbC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP¯ËãHûbC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP%9òéJübE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¾KòJýbE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPø™o NþbI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP rþJÿbE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP 9zN€bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPè‰2H¡bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPž]Y;¢b6 jet/accuracy_2_of_7"trainer/global_step2:JP ®øH£bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP³|£J¤bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPǨH¥bC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPwߪ&H¦bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP¯ ¬N§bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP”ûÛJ¨bE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPQ2š›N©bI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPd™+HªbC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPø¾¿J«bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPiœ0J¬bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP#oÛ!N­bI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPïs”\H®bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP7 Š›J¯bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPw‹.ÒN°bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP?4öænJÙbE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP«ÔÇNÚbI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPÑ õpNÛbI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP$­$ÛJÜbE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÍ6 ñ;Ýb6 jet/accuracy_3_of_4"trainer/global_step2:JPf‹;Þb6 jet/accuracy_4_of_4"trainer/global_step2:JPÜž“HßbC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPêBÊNàbI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPt•×JábE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‰PØûJâbE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPr#JãbE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPãCƒNäbI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP)>˜VHåbC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP„Ñ‹›HæbC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP>ÃJçbE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP}«pJèbE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÚÕ¦@éb; particle/accuracy_5_of_6"trainer/global_step2:JP?ôË›JêbE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP«ÃHëbC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP%dÛJìbE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPIÛ-NíbI &Purity/0FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPæC{~NîbI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPØÕ&dNïbI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP³ãg NðbI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP?ôÊOJñbE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPP¾%eHòbC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPíá=@JóbE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPðÅPHôbC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP[ÜÀNõbI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPj…¦éJöbE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP³Ö2úN÷bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP ‚Ï;øb6 jet/accuracy_2_of_5"trainer/global_step2:JPšç =GùbB validation_average_jet_accuracy"trainer/global_step2:JPFNÿðHúbC Purity/2FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPÌÃ6HûbC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP°m¾'NübI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPøA™NýbI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP`Î/…JþbE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP<>NÿbI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPõ6·N€ bI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPÛY'N bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPµBÈî;‚ b6 jet/accuracy_1_of_3"trainer/global_step2:JPpD''Hƒ bC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPÈ÷H„ bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPVYÎvJ… bE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP°0ÅbH† bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP¼†ïvJ‡ bE "Purity/3FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¤m2Hˆ bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPÛ^îËN‰ bI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPbä²–JŠ bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP jËhN‹ bI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP–CóAJŒ bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP!¯²J bE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPùUÿJŽ bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPjº“N bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP¼ÄJ bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPN-ÔN‘ bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP´ç¡ N’ bI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPƒ/DËH“ bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPû|€H” bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP²É¤‡J• bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¥µø0N– bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPŽþò¸N— bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPïeL!H˜ bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP9¬@™ b; particle/accuracy_1_of_5"trainer/global_step2:JP—#¨@Hš bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP|Ü J› bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPXdc¦Jœ bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPRÔÌPN bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPOéYªNž bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP3ÅäHŸ bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP„ çtJ  bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÆ9«N¡ bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPEU "N¢ bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPpYoH£ bC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÂô H¤ bC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPÏÙJ¥ bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP·T™›N¦ bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPâÜ­×N§ bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP¬ÆXJ¨ bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPå’)ÇN© bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP~Œ‰Hª bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP6ÖxH« bC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPrùiÕN¬ bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPœl"N­ bI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP®nººH® bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPN¥N¯ bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP ¯J° bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPô^éÜJ± bE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¼I”N² bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPžN³ bI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPX¥?¿N´ bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPxÉÞr@µ b; particle/accuracy_3_of_4"trainer/global_step2:JPÕÅ¥{J¶ bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPµmo^H· bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP—ã:=J¸ bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPh–zH¹ bC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPô@òLHº bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP.Ù¡.J» bE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP¯&·J¼ bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP´ÞN½ bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPŽ-à N¾ bI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPÿâÕN¿ bI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPß3!]HÀ bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JP©`ÌHÁ bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÁɲÚJ bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP>ŒO©Jà bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÆnŒ§HÄ bC Purity/2FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPê-èJÅ bE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPv‚•ªJÆ bE "Purity/0FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPäÒz¨NÇ bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP™ÆvNÈ bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP³¼¨ïHÉ bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPÌÈ àJÊ bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPï€ØNË bI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPôæäŽHÌ bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPc,® JÍ bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP³#$[NÎ bI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP JitHÏ bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPNN‘1JÐ bE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP”p‡JÑ bE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP%ɉôJÒ bE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP`îeJÓ bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPˆNÔ bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPëò¼ JÕ bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP²îÀ7HÖ bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP°Äš£N× bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPUÕ}¹HØ bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP‡hÓTNÙ bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP£©x#HÚ bC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPŒÖ?¹NÛ bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP'»¥NÜ bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP¼— ÈJÝ bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP%JÒ @Þ b; particle/accuracy_1_of_7"trainer/global_step2:JPéDJß bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPuC½Jà bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÀ?ªNá bI &Purity/4FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPP9ÞJâ bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPyç°0Jã bE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPgõýÌNä bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP(x®Hå bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP/wCoJæ bE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP±*ANç bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPl†Nè bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPoJC@é b; particle/accuracy_4_of_4"trainer/global_step2:JPù‡Ð_Nê bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPû·þJë bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP/z›3Hì bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP„ÑÿHŽ bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPy+H bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÈãÀS@ b; particle/accuracy_3_of_3"trainer/global_step2:JPÔË/ÇJ‘ bE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPy&.@’ b; particle/accuracy_2_of_7"trainer/global_step2:JPgÏDIJ“ bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPaógÊJ” bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÔ*¸cN• bI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP::"ÊJ– bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP|yf×H— bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP›šðJ˜ bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP#=«gN™ bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP äÙJš bE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPV‡-J› bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP«#œØHœ bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP\M5öJ bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP«ÍJž bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP`®_äHŸ bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPâ¶~uJ  bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP‡3pbJ¡ bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP2ßL—H¢ bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPe=¥‡H£ bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPfóõH¤ bC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP!J.H¥ bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPômu¬J¦ bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP‰p(¸N§ bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPvJóN¨ bI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP²¾©9© b4 particle/accuracy"trainer/global_step2:JP*ù†°Jª bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÖÁ?N« bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP †ëîN¬ bI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPÆ>ÉJ­ bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPÓ‚f;® b6 jet/accuracy_6_of_6"trainer/global_step2:JP½¡\H¯ bC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPň1@J° bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP¥oŽH± bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPÜŠí„N² bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP°¸ªH³ bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP覆a<´ b7 particle/sensitivity"trainer/global_step2:JPdž3—Hµ bC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP=”ÜœH¶ bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPŽgtºJ· bE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP =P÷J¸ bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPsK¾J¹ bE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP ²X;º b6 jet/accuracy_4_of_7"trainer/global_step2:JP™É$çN» bI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPAß×@H¼ bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPIsÒÙJ½ bE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¦[ çJ¾ bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPÂã4’J¿ bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÉÏ©NÀ bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPê)$0NÁ bI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP¦7‹XJ bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP£HONà bI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPvÞÒNÄ bI &Purity/4FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP ÄS·HÅ bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP¤øÿ•NÆ bI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPY1`|NÇ bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPavv·NÈ bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP5åî HÉ bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP¥}3¶HÊ bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP=$´Ô<Ë b7 particle/specificity"trainer/global_step2:JP#o¡JÌ bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPq-ÁJÍ bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP?è>HÎ bC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP>ËÎêHÏ bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP ÎhÒNÐ bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP´¬º2HÑ bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP [JHÒ bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPšš\NÓ bI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP”†MŒJÔ bE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP°HÕ bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÊæHÖ bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPtÀ{‰N× bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP‚7{‰JØ bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPø¶!NÙ bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPJ9ª+NÚ bI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPîÇCJÛ bE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¹«ÀÉHÜ bC Purity/1FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP7mæNÝ bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP6äùáJÞ bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP‚ÂáeHß bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPbÕ²ªHà bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÓG)äNá bI &Purity/0FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP2½ŒiJâ bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP{â@Hã bC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP«—­Jä bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPªFó¾Hå bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP:Þ.Hæ bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPªï‹Nç bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPû“ÞJè bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPªì'ÎNé bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPuuýHê bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPtVß)Hë bC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP!R¸Nì bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÿJí bE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP`¬ãkNî bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPEºµç;ï b6 jet/accuracy_1_of_1"trainer/global_step2:JP3ŒTÏJð bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP†Ë±Jñ bE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÃ8/Jò bE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP£ òJó bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP×FšUJô bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP›÷•9Hõ bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPOÊ=uJö bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP¬ƒ ŽN÷ bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPg¼h Nø bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP°ii‡Hù bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP‚>Jú bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP™UѾJû bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP; ïáNü bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPõË*ˆJý bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP6”>`Jþ bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPêXˆ8Hÿ bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP‰Ôê;€ b6 jet/accuracy_6_of_7"trainer/global_step2:JPɦ LJ bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPjÕ2GJ‚ bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP‚à_DHƒ bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JP 4H„ bC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPþ\N… bI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPLe(EJ† bE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPc{Ú§J‡ bE "Purity/1FRt1SRqqt4FBt/event_purity"trainer/global_step2:JP.¬ˆMNˆ bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP$?jON‰ bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP“~¼ÐNŠ bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPÒÆ*4J‹ bE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP\>ì+JŒ bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP0¹VN bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP@°f#JŽ bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPqK+¦J bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP åÖ>@ b; particle/accuracy_3_of_5"trainer/global_step2:JPCék J‘ bE "Purity/3FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPo­óª@’ b; particle/accuracy_5_of_7"trainer/global_step2:JPëÂú{J“ bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP8÷N” bI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP®–ÛN• bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP,Ë9N– bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP׊îí;— b6 jet/accuracy_1_of_4"trainer/global_step2:JP‡³¹3J˜ bE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPý( Þ;™ b6 jet/accuracy_1_of_7"trainer/global_step2:JP™ãŠ Hš bC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP›£ÜYN› bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP®R““;œ b6 jet/accuracy_2_of_6"trainer/global_step2:JP‘¾âíJ bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP£.Hž bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPê¿3nJŸ bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP†)-J  bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPHŽ0¡N¡ bI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÙ/SJ¢ bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP%_é@£ b; particle/accuracy_4_of_6"trainer/global_step2:JP \@¤ b; particle/accuracy_6_of_7"trainer/global_step2:JP?-ÃÇN¥ bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPЦ­J¦ bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPûÆnJ§ bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP$_ÍœH¨ bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÃt'H© bC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JP§í+Nª bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPIºWN« bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPñ'ß@¬ b; particle/accuracy_1_of_3"trainer/global_step2:JP>jÔJ­ bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPggΆJ® bE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP jä¦H¯ bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPC6G(J° bE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÈdߘN± bI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPŸ³÷N² bI &Purity/3FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPöjOH³ bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP¾Áý†J´ bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPæv¹Hµ bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÐŽ0N¶ bI &Purity/0FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPö;™H· bC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPB"MÖNƒ bI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP~zÞN„ bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÜ (‘H… bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP‚«Ž‡J† bE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP#Á/º8‡ b3 particle/f_score"trainer/global_step2:JPeQ;ˆ b6 jet/accuracy_5_of_7"trainer/global_step2:JP"§)´H‰ bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPgÁÍNŠ bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPŠ|ƒ¯H‹ bC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPý*HŒ bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÒF¾(N bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÑòµ²NŽ bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPø竧H bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP„R©ÁN bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPlö^@‘ b; particle/accuracy_4_of_5"trainer/global_step2:JP˜…?J’ bE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPQ¾Z>H“ bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPmkc H” bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP; ~H• bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP«’ÆÅH– bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP«2ˆßH— bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÁœ7œJ˜ bE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP!f¨ÁN™ bI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP™>åHš bC Purity/3FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP<~J› bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÃ$ßZHœ bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPÝCbBH bC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPXÃ9`Hž bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP'>åßHŸ bC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPÔäпJ  bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP=Ž¯N¡ bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPñSÏJ¢ bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPë ÷îN£ bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP}HärJ¤ bE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPuHÖ@H¥ bC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP®<~J¦ bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPª¨ì™J§ bE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPPØBsN¨ bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPäH»kJ© bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP }¾Nª bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP•3N« bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPe‹ç;¬ b6 jet/accuracy_1_of_5"trainer/global_step2:JPÿ££+H­ bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP—{|H® bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP¸ã‹£J¯ bE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÉ.#J° bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP, jJ± bE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP˜#ñH² bC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPn3–‘J³ bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPDYþÝH´ bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPÿÎ;Hµ bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP’g€H¶ bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP-9QE2· - _wandbruntime‚1179  _runtime‚1179EMÅ;¸ ¥ 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9684422612190247 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.22511306405067444 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9857746362686157 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11205846816301346 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9885057210922241 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3117816150188446 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1288580298423767 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3866666853427887 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 trainer/global_step‚1208 _runtime‚1179.543301036 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3039649724960327 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3437499701976776 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.06060606241226196 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.31753814220428467 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3121010661125183 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8706285357475281 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.375741571187973 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0044143651612102985 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.40221819281578064 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.09583708643913269 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.2800000011920929 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14526765048503876 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9979325532913208 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1150793582201004 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2788146734237671 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9893184304237366 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ;"Purity/1FRt*SRqqt1FBt/event_purity‚0.019763419404625893 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.8653846383094788 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 (loss/val_total_loss‚54.7792555439962 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09876542538404465 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.24593359231948853 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2678571343421936 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.032295893877744675 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2563062906265259 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.04266221076250076 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01166001707315445 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2222222238779068 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08871713280677795 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.41867050528526306 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.07695433497428894 jet/accuracy_2_of_7‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9783612489700317 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3342468738555908 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1493566334247589 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.24259261786937714 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.37859392166137695 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_6‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_3_of_5‚0.023969532921910286 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.203125 particle/accuracy_3_of_7‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.05978389456868172 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09629856050014496 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 +jet/accuracy_3_of_6‚0.01923076994717121 jet/accuracy_4_of_6‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_7_of_7‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2951388955116272 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.30000001192092896 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2083333432674408 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1295674443244934 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9921548366546631 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1511494219303131 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_4‚0.015389222651720047 jet/accuracy_4_of_4‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3159644305706024 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2601364552974701 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.19126981496810913 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14681929349899292 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.31196942925453186 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.553030252456665 particle/accuracy_5_of_6‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.031101105734705925 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9806742668151855 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2582375705242157 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 *jet/accuracy_2_of_5‚0.2685803771018982 6validation_average_jet_accuracy‚0.3473295271396637 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3503584563732147 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7478393316268921 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07201013714075089 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.2368421107530594 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2083333432674408 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6075806021690369 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 <"Purity/1FRt1SRqqt1FBt/event_purity‚0.0011337868636474013 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.306990385055542 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13564924895763397 particle/accuracy_1_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.34126654267311096 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.16018518805503845 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10105083882808685 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2619929313659668 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4295634627342224 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15789473056793213 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.06444444507360458 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.07941197603940964 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9906432628631592 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.2982509732246399 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4177122712135315 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.11792928725481033 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.26688915491104126 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.295678973197937 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07565538585186005 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.20552054047584534 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09674351662397385 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.35321593284606934 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 6 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.993154764175415 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9836652874946594 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.23719944059848785 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.03832944855093956 ,&Purity/4FRt1SRqqt1FBt/evenÖ aÃVt_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9879561066627502 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.27932098507881165 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.35284706950187683 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9855875372886658 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4108039140701294 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2976591885089874 5 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.92241370677948 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4046245217323303 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.24434593319892883 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9934678077697754 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.28400668501853943 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11623696237802505 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 jet/accuracy_5_of_6‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.25171130895614624 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3889329731464386 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8057171106338501 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13143834471702576 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9928571581840515 particle/accuracy_3_of_3‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 particle/accuracy_2_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3348199427127838 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3253411054611206 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.26356589794158936 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9811714291572571 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.01252052467316389 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.553030252456665 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9978840351104736 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3586485683917999 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9664648175239563 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 'particle/accuracy‚0.822451651096344 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.06164246052503586 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,particle/sensitivity‚0.17597290873527527 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.42648810148239136 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3328147232532501 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.34041863679885864 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.32643628120422363 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.30466127395629883 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.14278370141983032 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 +particle/specificity‚0.9420652985572815 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3480674624443054 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13359622657299042 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.994266927242279 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9579861760139465 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2905701696872711 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09431347250938416 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9784773588180542 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.31742915511131287 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6049950122833252 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.33579325675964355 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9844042062759399 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3160439133644104 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14600855112075806 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977535605430603 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11772800981998444 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *jet/accuracy_1_of_1‚0.3945295810699463 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.26543208956718445 ,"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.375 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.39151662588119507 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9458717107772827 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.013326149433851242 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.11193692684173584 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.35284706950187683 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.2222222238779068 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 _step‚98 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0019841270986944437 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.416202574968338 particle/accuracy_3_of_5‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 particle/accuracy_5_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_4‚0.7953428030014038 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 +jet/accuracy_2_of_6‚0.21153846383094788 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.008352534845471382 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.18954813480377197 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.03514424338936806 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_6‚1 particle/accuracy_6_of_7‚0.5 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 epoch‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.14459766447544098 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 particle/accuracy_1_of_3‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3777298331260681 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.2625391483306885 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.181547611951828 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.6297249794006348 *"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_7‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_3‚0.13012470304965973 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.18198877573013306 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42508211731910706 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.029558856040239334 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0036584853660315275 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 particle/accuracy_2_of_5‚1 particle/accuracy_6_of_6‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08539210259914398 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08683854341506958 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1078445166349411 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9909297227859497 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.36958280205726624 ) Purity/1FRt0SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9873145818710327 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.31535857915878296 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.05089564621448517 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08749798685312271 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.18298780918121338 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.27803686261177063 particle/accuracy_1_of_4‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3836805522441864 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1354299783706665 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9948887228965759 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.980555534362793 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1265016794204712 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_3‚0.008017250336706638 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.27911239862442017 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3677641749382019 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.34539446234703064 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.43536922335624695 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.22727273404598236 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.2271091789007187 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.2647058963775635 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9767441749572754 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9895525574684143 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2796870470046997 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9725949168205261 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1072796881198883 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.20000000298023224 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4050179123878479 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.19930554926395416 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9712962508201599 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08865249156951904 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ;"Purity/1FRt0SRqqt2FBt/event_purity‚0.050513919442892075 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1303466558456421 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9941308498382568 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.295473575592041 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9978840351104736 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3014184236526489 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.17442528903484344 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.25993186235427856 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_2‚1 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3129562437534332 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 particle/accuracy_2_of_4‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2361111044883728 particle/accuracy_1_of_6‚1 particle/accuracy_2_of_6‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07201013714075089 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.31774547696113586 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30916187167167664 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.004716981202363968 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9827523231506348 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9856962561607361 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.23348580300807953 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.30446431040763855 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03695160895586014 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14058761298656464 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.39287373423576355 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08959061652421951 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15438595414161682 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2723875641822815 (particle/f_score‚0.23621664941310883 jet/accuracy_5_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 $ _timestamp‚1.788417708791179e+099ÎJ/)¹ …¥ A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6049950122833252 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.33579325675964355 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9844042062759399 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3160439133644104 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14600855112075806 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977535605430603 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11772800981998444 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *jet/accuracy_1_of_1‚0.3945295810699463 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.26543208956718445 ,"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.375 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.39151662588119507 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9458717107772827 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.013326149433851242 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.11193692684173584 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.35284706950187683 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.2222222238779068 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 _step‚98 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0019841270986944437 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.416202574968338 particle/accuracy_3_of_5‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 particle/accuracy_5_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_4‚0.7953428030014038 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 +jet/accuracy_2_of_6‚0.21153846383094788 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.008352534845471382 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.18954813480377197 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.03514424338936806 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_6‚1 particle/accuracy_6_of_7‚0.5 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 epoch‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.14459766447544098 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 particle/accuracy_1_of_3‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3777298331260681 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.2625391483306885 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.181547611951828 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.6297249794006348 *"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_7‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_3‚0.13012470304965973 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.18198877573013306 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42508211731910706 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.029558856040239334 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0036584853660315275 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 particle/accuracy_2_of_5‚1 particle/accuracy_6_of_6‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08539210259914398 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08683854341506958 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1078445166349411 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9909297227859497 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.36958280205726624 ) Purity/1FRt0SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9873145818710327 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.31535857915878296 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.05089564621448517 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08749798685312271 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.18298780918121338 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.27803686261177063 particle/accuracy_1_of_4‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3836805522441864 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1354299783706665 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9948887228965759 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.980555534362793 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1265016794204712 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_3‚0.008017250336706638 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.27911239862442017 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3677641749382019 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.34539446234703064 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.43536922335624695 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.22727273404598236 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.2271091789007187 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.2647058963775635 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9767441749572754 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9895525574684143 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2796870470046997 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9725949168205261 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1072796881198883 8 Purity/*FRt3SRqqt1FBt/FRt_†µ$]ipurity‚0.20000000298023224 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4050179123878479 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.19930554926395416 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9712962508201599 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08865249156951904 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ;"Purity/1FRt0SRqqt2FBt/event_purity‚0.050513919442892075 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1303466558456421 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9941308498382568 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.295473575592041 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9978840351104736 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3014184236526489 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.17442528903484344 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.25993186235427856 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_2‚1 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3129562437534332 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 particle/accuracy_2_of_4‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2361111044883728 particle/accuracy_1_of_6‚1 particle/accuracy_2_of_6‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07201013714075089 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.31774547696113586 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30916187167167664 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.004716981202363968 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9827523231506348 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9856962561607361 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.23348580300807953 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.30446431040763855 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03695160895586014 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14058761298656464 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.39287373423576355 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08959061652421951 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15438595414161682 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2723875641822815 (particle/f_score‚0.23621664941310883 jet/accuracy_5_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 $ _timestamp‚1.788417708791179e+09 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9684422612190247 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.22511306405067444 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9857746362686157 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11205846816301346 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9885057210922241 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3117816150188446 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1288580298423767 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3866666853427887 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 trainer/global_step‚1208 _runtime‚1179.543301036 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3039649724960327 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3437499701976776 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.06060606241226196 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.31753814220428467 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3121010661125183 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8706285357475281 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.375741571187973 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0044143651612102985 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.40221819281578064 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.09583708643913269 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.2800000011920929 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14526765048503876 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9979325532913208 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1150793582201004 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2788146734237671 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9893184304237366 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ;"Purity/1FRt*SRqqt1FBt/event_purity‚0.019763419404625893 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.8653846383094788 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 (loss/val_total_loss‚54.7792555439962 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09876542538404465 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.24593359231948853 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2678571343421936 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.032295893877744675 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2563062906265259 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.04266221076250076 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01166001707315445 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2222222238779068 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08871713280677795 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.41867050528526306 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.07695433497428894 jet/accuracy_2_of_7‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9783612489700317 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3342468738555908 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1493566334247589 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.24259261786937714 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.37859392166137695 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_6‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_3_of_5‚0.023969532921910286 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.203125 particle/accuracy_3_of_7‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.05978389456868172 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09629856050014496 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 +jet/accuracy_3_of_6‚0.01923076994717121 jet/accuracy_4_of_6‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_7_of_7‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2951388955116272 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.30000001192092896 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2083333432674408 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1295674443244934 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9921548366546631 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1511494219303131 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_4‚0.015389222651720047 jet/accuracy_4_of_4‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3159644305706024 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2601364552974701 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.19126981496810913 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14681929349899292 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.31196942925453186 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.553030252456665 particle/accuracy_5_of_6‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.031101105734705925 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9806742668151855 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2582375705242157 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 *jet/accuracy_2_of_5‚0.2685803771018982 6validation_average_jet_accuracy‚0.3473295271396637 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3503584563732147 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7478393316268921 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07201013714075089 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.2368421107530594 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2083333432674408 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6075806021690369 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 <"Purity/1FRt1SRqqt1FBt/event_purity‚0.0011337868636474013 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.306990385055542 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13564924895763397 particle/accuracy_1_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.34126654267311096 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.16018518805503845 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10105083882808685 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2619929313659668 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4295634627342224 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15789473056793213 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.06444444507360458 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.07941197603940964 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9906432628631592 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.2982509732246399 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4177122712135315 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.11792928725481033 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.26688915491104126 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.295678973197937 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07565538585186005 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.20552054047584534 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09674351662397385 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.35321593284606934 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 6 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.993154764175415 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9836652874946594 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.23719944059848785 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.03832944855093956 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9879561066627502 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.27932098507881165 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.35284706950187683 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9855875372886658 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4108039140701294 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2976591885089874 5 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.92241370677948 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4046245217323303 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.24434593319892883 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9934678077697754 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.28400668501853943 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11623696237802505 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 jet/accuracy_5_of_6‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.25171130895614624 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3889329731464386 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8057171106338501 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13143834471702576 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9928571581840515 particle/accuracy_3_of_3‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 particle/accuracy_2_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3348199427127838 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3253411054611206 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.26356589794158936 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9811714291572571 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.01252052467316389 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.553030252456665 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9978840351104736 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3586485683917999 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9664648175239563 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 'particle/accuracy‚0.822451651096344 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.06164246052503586 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,particle/sensitivity‚0.17597290873527527 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.42648810148239136 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3328147232532501 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.34041863679885864 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.32643628120422363 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.30466127395629883 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.14278370141983032 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 +particle/specificity‚0.9420652985572815 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3480674624443054 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13359622657299042 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.994266927242279 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9579861760139465 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2905701696872711 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09431347250938416 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9784773588180542 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.31742915511131287 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875b8(}÷º ‚(:… ²­äÔ«Ûœx gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚124.038' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8076 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö[­» ‚(: ²­äÔí‘Ñz disk.rbd2.in‚0 proc.memory.rssMB‚ 2842.0625* proc.memory.percent‚17.346572875976562 disk.sda2.in‚0 disk.rbd3.out‚0# memory_percent‚97.8994369506836' disk.nvme0c0n1.out‚4753.89306640625% disk.nvme0n1.out‚4753.89306640625 disk.dm-0.in‚4 disk.rbd0.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚4 disk.dm-1.out‚42.46875 disk.rbd0.in‚0 cpu‚25.28587029183329 disk.sda.out‚42.46875 disk.rbd1.out‚ 0.55029296875 disk.drbd1777.in‚0 disk.rbd4.out‚0 network.sent‚ 1.677115e+06$ disk./.usageGB‚1096.936134338379" disk.dm-0.out‚4753.89306640625 disk.dm-1.in‚0) disk./.usagePercent‚73.61630436616959 disk.sda3.in‚0 disk.sda2.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚4 disk.rbd3.in‚0% proc.memory.availableMB‚ 344.15625 network.recv‚473821 disk.sda3.out‚42.46875 disk.rbd2.out‚ 74.95703125 disk.sda1.out‚0ö9¼ ‚(:… Á­äÔ¶Ýx gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.936' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\Øõ‰(½ ‚(: Á­äÔÉÛ¤{ disk.rbd0.out‚0$ memory_percent‚97.89590835571289 disk.rbd0.in‚0 network.sent‚ 1.800947e+06 network.recv‚480508% disk./.usageGB‚1096.9362335205078% disk.nvme0n1.out‚5071.35302734375 disk.drbd1777.out‚0) disk./.usagePercent‚73.61631102236548 disk.dm-1.out‚ 43.01953125 disk.rbd2.in‚0 disk.rbd4.out‚0' disk.nvme0c0n1.out‚5071.35302734375 disk.sda3.in‚0 disk.dm-0.in‚4& proc.memory.availableMB‚ 344.734375# proc.memory.rssMB‚ 2841.06640625* proc.memory.percent‚17.340493202209473 cpu‚25.282753584721497 disk.sda.in‚0 disk.sda.out‚ 43.01953125 disk.sda3.out‚ 43.01953125 disk.rbd1.in‚0 disk.nvme0n1.in‚4 disk.rbd2.out‚76.46875 disk.sda1.out‚0 disk.rbd3.out‚0" disk.dm-0.out‚5071.35302734375 disk.sda2.out‚0 disk.nvme0c0n1.in‚4 proc.cpu.threads‚17 disk.rbd1.out‚ 0.55029296875 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.rbd4.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0Yô–†¾ ‚(:„ ЭäÔà¬x gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.623 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÙ L¿ ‚(:’ ЭäÔΖâz% disk./.usageGB‚1096.7674026489258 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd4.in‚0$ disk.nvme0c0n1.out‚ 5073.70703125 disk.rbd3.in‚0" disk.nvme0n1.out‚ 5073.70703125 disk.sda2.out‚0) disk./.usagePercent‚73.60498064091506 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.dm-0.out‚ 5073.70703125 disk.dm-1.out‚ 43.5078125 proc.cpu.threads‚17 network.sent‚ 1.806172e+06 network.recv‚483796 disk.sda.in‚0 disk.sda3.out‚ 43.5078125 disk.dm-0.in‚4( proc.memory.availableMB‚ 344.70703125 cpu‚25.279963628037923 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 43.5078125 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4 disk.rbd1.out‚ 0.61328125 disk.dm-1.in‚0$ memory_percent‚97.89607524871826# proc.memory.rssMB‚ 2841.06640625 disk.nvme0n1.in‚4 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd2.out‚ 76.6328125 disk.rbd1.in‚0* proc.memory.percent‚17.340493202209473~P¿¹2À - _wandbruntime‚1219  _runtime‚1219ëUDÅÁ ¿ & lr-AdamW/pg1‚0.0008022919185687065 & lr-AdamW/pg2‚0.0008022919185687065 trainer/global_step‚1249 $ _timestamp‚1.788417748339157e+09 _runtime‚1219.089165994 _step‚99yioYÉ Ã & lr-AdamW/pg1‚0.0008022919185687065 & lr-AdamW/pg2‚0.0008022919185687065 trainer/global_step‚1249 $ _timestamp‚1.788417748339157e+09 _runtime‚1219.089165994 _step‚99c '&ó2à - _wandbruntime‚1219  _runtime‚1219K¥»%OÄ ¿O QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2321362910208 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51907481595904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23101871 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53203781534720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2314395306343 % _timestamp‚1.7884177483399246e+09 O@DeviceStatsMonitor.Xr$Äv$on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292762624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32533739 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55571773359104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2367991824384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1219.090107428 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51907481595904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8445009 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55635610 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2321362910208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2367723001856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46950728602624 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 _step‚100 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23101871 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55571480596480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2321134707712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2314618955483 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 331676672 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46950711563264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2321134707712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268822528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 trainer/global_step‚1249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13519754 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51907727801344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32559407 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 290068992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0-¢½ðÉ'Å ÃO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292762624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32533739 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55571773359104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2367991824384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1219.090107428 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51907481595904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8445009 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55635610 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2321362910208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2367723001856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46950728602624 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 _step‚100 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23101871 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55571480596480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2321134707712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2314618955483 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 331676672 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46950711563264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2321134707712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268822528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 trainer/global_step‚1249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13519754 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51907727801344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32559407 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 290068992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2321362910208 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51907481595904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23101871 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53203781534720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2314395306343 % _timestamp‚1.7884177483399246e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428d–¾,2Æ - _wandbruntime‚1219  _runtime‚1219›&[ŽÏ&Ç ÉM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13530109 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301137920 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32584984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 trainer/global_step‚1249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49624684832256 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2322972402176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49624702835200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚49624702835200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2323200617984 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46987031060480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32559290 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49303482415771 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23119590 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51947903453184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55704576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32559290 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55678880 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55614814121984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53244927290880 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13530107 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51947903453184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49624684832256 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55704576 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55614512984064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2323200617984 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32584984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 331676672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23119590 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 290068992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21980493 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53244951230976 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚101 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8450386 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2316451355291 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55678880 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚49303241727063 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2322972402176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 _runtime‚1219.946052542 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚01(ÌO È ÍM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49624702835200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚49624702835200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2323200617984 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46987031060480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32559290 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49303482415771 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23119590 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51947903453184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55704576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32559290 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55678880 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55614814121984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53244927290880 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13530107 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51947903453184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49624684832256 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55704576 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55614512984064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2323200617984 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32584984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 331676672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23119590 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 290068992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21980493 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53244951230976 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚101 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8450386 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2316451355291 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55678880 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚49303241727063 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2322972402176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 _runtime‚1219.946052542 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13530109 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301137920 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32584984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 trainer/global_step‚1249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49624684832256 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2322972402176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0es—2É - _wandbruntime‚1219  _runtime‚1219$‚,ÌZÊ Ô /loss/FRt3/assignment_loss‚4.212806700513913 2loss/SRqqt3/assignment_loss‚1.8367253023645151 /loss/FRt3/detection_loss‚2.4785256488965106 1loss/FBt2/assignment_loss‚0.24020865981180708 2loss/SRqqt4/assignment_loss‚2.3503701406029553 % _timestamp‚1.7884177491970952e+09 _runtime‚1219.946941927 /loss/FRt4/assignment_loss‚4.438920071721077 0loss/FBt3/assignment_loss‚0.6419978141767615 1loss/FBt4/assignment_loss‚0.34286664597053307 1loss/SRqqt2/detection_loss‚1.1275823985536892 /loss/FBt3/detection_loss‚5.2677857875823975 epoch‚1 0loss/FBt1/assignment_loss‚0.7290414969902486 /loss/FRt1/detection_loss‚2.4449759662844412 1loss/SRqqt1/detection_loss‚2.4514648130065515 1loss/SRqqt4/detection_loss‚2.1003455359202166 /loss/FBt2/detection_loss‚0.8186199544844303 /loss/FRt1/assignment_loss‚4.399748749511186 /loss/FRt2/assignment_loss‚2.398780065424302 .loss/FBt1/detection_loss‚2.525938630104065 /loss/FBt4/detection_loss‚3.1530412673950194 trainer/global_step‚1249 2loss/SRqqt1/assignment_loss‚2.2664711208719956 2loss/SRqqt2/assignment_loss‚1.9223504696901028 .loss/FRt4/detection_loss‚2.593109244853258 1loss/SRqqt3/detection_loss‚1.9902654465125955 /loss/FRt2/detection_loss‚1.8992957414365281 $loss/total_loss‚54.6312376726791 _step‚102WˆtD^Ë Ø 2loss/SRqqt1/assignment_loss‚2.2664711208719956 2loss/SRqqt2/assignment_loss‚1.9223504696901028 .loss/FRt4/detection_loss‚2.593109244853258 1loss/SRqqt3/detection_loss‚1.9902654465125955 /loss/FRt2/detection_loss‚1.8992957414365281 $loss/total_loss‚54.6312376726791 _step‚102 /loss/FRt3/assignment_loss‚4.212806700513913 2loss/SRqqt3/assignment_loss‚1.8367253023645151 /loss/FRt3/detection_loss‚2.4785256488965106 1loss/FBt2/assignment_loss‚0.24020865981180708 2loss/SRqqt4/assignment_loss‚2.3503701406029553 % _timestamp‚1.7884177491970952e+09 _runtime‚1219.946941927 /loss/FRt4/assignment_loss‚4.438920071721077 0loss/FBt3/assignment_loss‚0.6419978141767615 1loss/FBt4/assignment_loss‚0.34286664597053307 1loss/SRqqt2/detection_loss‚1.1275823985536892 /loss/FBt3/detection_loss‚5.2677857875823975 epoch‚1 0loss/FBt1/assignment_loss‚0.7290414969902486 /loss/FRt1/detection_loss‚2.4449759662844412 1loss/SRqqt1/detection_loss‚2.4514648130065515 1loss/SRqqt4/detection_loss‚2.1003455359202166 /loss/FBt2/detection_loss‚0.8186199544844303 /loss/FRt1/assignment_loss‚4.399748749511186 /loss/FRt2/assignment_loss‚2.398780065424302 .loss/FBt1/detection_loss‚2.525938630104065 /loss/FBt4/detection_loss‚3.1530412673950194 trainer/global_step‚1249fábHÌ ‚(:„ ß­äÔ誓x gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.476' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06¦7ðÍ ‚(: ß­äÔ¤˜Ñz disk.rbd1.in‚0 disk.nvme0n1.in‚4 disk.rbd0.out‚0 disk.sda.out‚ 43.859375# memory_percent‚97.9069471359253 disk.dm-1.in‚0 disk.rbd2.out‚ 78.14453125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 342.92578125* proc.memory.percent‚17.340493202209473 cpu‚25.27722370127014) disk./.usagePercent‚73.60487004566028 disk.drbd1777.in‚0 disk.sda3.out‚ 43.859375# proc.memory.rssMB‚ 2841.06640625$ disk./.usageGB‚1096.765754699707 disk.rbd1.out‚ 0.61328125 disk.dm-0.in‚4 disk.rbd2.in‚0$ disk.nvme0c0n1.out‚ 5074.10546875 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 43.859375 disk.rbd4.out‚0 disk.rbd3.in‚0 network.recv‚488068 disk.dm-0.out‚ 5074.10546875 disk.sda2.in‚0 disk.nvme0c0n1.in‚4 disk.rbd3.out‚0 network.sent‚ 1.812053e+06" disk.nvme0n1.out‚ 5074.10546875 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 proc.cpu.threads‚17°€ÏÎ ‚(:… î­äÔ½ Éx gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.322' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½-Ï ‚(:¢ î­äÔõî„{ network.recv‚494321' disk.nvme0c0n1.out‚5208.38623046875 disk.rbd3.out‚0 disk.sda3.in‚0" disk.dm-0.out‚5208.38623046875 disk.dm-1.in‚0 network.sent‚ 1.920255e+06 disk.rbd3.in‚0 disk.dm-0.in‚4 disk.sda2.out‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0) disk./.usagePercent‚73.61359759851005 disk.sda1.in‚0 disk.sda.out‚ 44.50390625 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2841.06640625 disk.sda.in‚0% disk.nvme0n1.out‚5208.38623046875( proc.memory.availableMB‚ 342.52734375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 44.50390625 disk.rbd4.out‚0* proc.memory.percent‚17.340493202209473 cpu‚25.274526151775234 disk.rbd0.in‚0 disk.rbd1.out‚ 0.67626953125 disk.rbd2.out‚ 79.88671875$ memory_percent‚97.90937900543213% disk./.usageGB‚1096.8958015441895 disk.sda3.out‚ 44.50390625 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4 disk.nvme0n1.in‚4îÆ6¦›Ð ‚(: ý­äÔ㤓x gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04¿*Ñ ‚(:Ÿ ý­äÔÚŒ¼{ disk.sda.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2841.06640625 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.67626953125 disk.sda2.in‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 8.02734375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.dm-0.out‚5341.77294921875 network.sent‚ 1.926605e+06% disk./.usageGB‚1097.0249366760254 disk.rbd0.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚4' disk.nvme0c0n1.out‚5341.77294921875 network.recv‚499114 disk.sda3.out‚ 44.80859375 disk.nvme0n1.in‚4 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0* proc.memory.percent‚17.340493202209473) disk./.usagePercent‚73.62226396555917 disk.sda1.in‚0 disk.dm-0.in‚4 disk.rbd2.out‚ 79.88671875 disk.sda.out‚ 44.80859375 disk.rbd1.in‚0% disk.nvme0n1.out‚5341.77294921875 disk.dm-1.out‚ 44.80859375$ memory_percent‚99.95100498199463 cpu‚25.27189122264511뾇 2Ò - _wandbruntime‚1264  _runtime‚1264(†ÇÓ Á % _timestamp‚1.7884177938767307e+09 _runtime‚1264.626676317 _step‚103 & lr-AdamW/pg1‚0.0008021832426989753 & lr-AdamW/pg2‚0.0008021832426989753 trainer/global_step‚1299g˜Ó¯ËÔ Å _step‚103 & lr-AdamW/pg1‚0.0008021832426989753 & lr-AdamW/pg2‚0.0008021832426989753 trainer/global_step‚1299 % _timestamp‚1.7884177938767307e+09 _runtime‚1264.626676317gí2Õ - _wandbruntime‚1264  _runtime‚12645brÅ'Ö ¿O VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307442688 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48765851495424 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51503244091392 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2413033844736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23987821 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2413262047232 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8767737 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22805241 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51171863333615 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚48765834456064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51503262094336 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2406252526691 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53916277936128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22804569 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23987823 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2406028877551 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53916524141568 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚1299 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53916277936128 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57828132 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14037504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51172104022115 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23987823 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23987821 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884177938775797e+09 _runtime‚1264.628015749 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2413262047232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57802462 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 283502592 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51503244091392 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33814641 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14037502 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51503262094336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8767067 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243K1²ßŒ× ÃO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51503244091392 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33814641 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14037502 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51503262094336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8767067 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307442688 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48765851495424 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51503244091392 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2413033844736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23987821 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2413262047232 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8767737 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22805241 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51171863333615 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚48765834456064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51503262094336 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2406252526691 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53916277936128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22804569 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23987823 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2406028877551 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53916524141568 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚1299 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53916277936128 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57828132 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14037504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51172104022115 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23987823 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23987821 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884177938775797e+09 _runtime‚7Œ}=1264.628015749 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2413262047232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57802462 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 283502592 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22821152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51209998191071 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14047857 _runtime‚1265.533484306 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24005540 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53956699793408 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53956453574656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57845732 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8773295 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296943616 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51210238879779 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22821783 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14047859 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 trainer/global_step‚1299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2414871539200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55303436043776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884177947828856e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚53956699793408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24005540 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2463286799872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33865886 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55303412103680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2463013796352 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57871428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57766722843648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚105 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57845732 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51541582035456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48802153953280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51541582035456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33840192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2414871539200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2408084926499 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48802136913920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2415099755008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57766425900032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884177947828856e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚53956699793408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24005540 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2463286799872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33865886 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55303412103680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2463013796352 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57871428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57766722843648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚105 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57845732 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51541582035456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48802153953280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51541582035456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33840192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2414871539200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2408084926499 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48802136913920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2415099755008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57766425900032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22821152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51209998191071 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14047857 _runtime‚1265.533484306 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24005540 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53956699793408 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53956453574656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57845732 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8773295 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296943616 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51210238879779 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22821783 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14047859 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 trainer/global_step‚1299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752i6÷X62Û - _wandbruntime‚1265  _runtime‚1265¤‡åUWÜ Ñ 1loss/SRqqt3/assignment_loss‚2.256657389077273 /loss/FBt4/assignment_loss‚0.640607038885355 /loss/FRt4/detection_loss‚2.6194165102073126 1loss/SRqqt2/detection_loss‚1.3617470885316532 epoch‚1 _step‚106 .loss/FRt2/assignment_loss‚2.62162996311577 0loss/FBt1/assignment_loss‚0.7132987216703441 trainer/global_step‚1299 2loss/SRqqt1/assignment_loss‚2.3501362688839436 2loss/SRqqt4/assignment_loss‚2.0494733275251185 1loss/FBt2/assignment_loss‚0.20937082859017098 0loss/FBt3/assignment_loss‚0.6591154038906097 /loss/FBt4/detection_loss‚2.7386296808719637 /loss/FRt4/assignment_loss‚4.450559186935425 2loss/SRqqt2/assignment_loss‚1.8322035626818736 0loss/SRqqt3/detection_loss‚1.828179669380188 % _timestamp‚1.7884177947848096e+09 /loss/FRt1/assignment_loss‚4.093392032565492 .loss/FRt1/detection_loss‚2.928497010108196 /loss/FRt2/detection_loss‚1.8339633795679833 /loss/FRt3/detection_loss‚2.5230797307435857 &loss/total_loss‚52.587265223873416 0loss/FRt3/assignment_loss‚4.0177208601042285 0loss/SRqqt4/detection_loss‚2.222466620993107 /loss/FBt2/detection_loss‚0.8262614858074065 .loss/FBt3/detection_loss‚3.335937187075615 _runtime‚1265.534636656 1loss/SRqqt1/detection_loss‚2.5505837108939886 /loss/FBt1/detection_loss‚1.9243385657668113ÖÚag[Ý Õ _step‚106 .loss/FRt2/assignment_loss‚2.62162996311577 0loss/FBt1/assignment_loss‚0.7132987216703441 trainer/global_step‚1299 2loss/SRqqt1/assignment_loss‚2.3501362688839436 2loss/SRqqt4/assignment_loss‚2.0494733275251185 1loss/FBt2/assignment_loss‚0.20937082859017098 0loss/FBt3/assignment_loss‚0.6591154038906097 /loss/FBt4/detection_loss‚2.7386296808719637 /loss/FRt4/assignment_loss‚4.450559186935425 2loss/SRqqt2/assignment_loss‚1.8322035626818736 0loss/SRqqt3/detection_loss‚1.828179669380188 % _timestamp‚1.7884177947848096e+09 /loss/FRt1/assignment_loss‚4.093392032565492 .loss/FRt1/detection_loss‚2.928497010108196 /loss/FRt2/detection_loss‚1.8339633795679833 /loss/FRt3/detection_loss‚2.5230797307435857 &loss/total_loss‚52.587265223873416 0loss/FRt3/assignment_loss‚4.0177208601042285 0loss/SRqqt4/detection_loss‚2.222466620993107 /loss/FBt2/detection_loss‚0.8262614858074065 .loss/FBt3/detection_loss‚3.335937187075615 _runtime‚1265.534636656 1loss/SRqqt1/detection_loss‚2.5505837108939886 /loss/FBt1/detection_loss‚1.9243385657668113 1loss/SRqqt3/assignment_loss‚2.256657389077273 /loss/FBt4/assignment_loss‚0.640607038885355 /loss/FRt4/detection_loss‚2.6194165102073126 1loss/SRqqt2/detection_loss‚1.3617470885316532 epoch‚1jZ§áI›Þ ‚(: Œ®äÔìä²x gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.156' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.031200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í«¬ ß ‚(:‘ Œ®äÔ‚Þú{ disk.rbd2.out‚81.5625 cpu‚25.269313666473856" disk.nvme0n1.out‚ 5342.38671875 disk.dm-1.out‚ 45.54296875 proc.cpu.threads‚17 disk.rbd1.out‚ 0.67626953125 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚4 disk.sda1.out‚0 disk.rbd3.out‚0* proc.memory.percent‚17.340493202209473) disk./.usagePercent‚73.62234230386464 disk.dm-0.out‚ 5342.38671875 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd4.out‚0$ proc.memory.availableMB‚7.390625 network.sent‚ 1.931811e+06 disk.sda3.out‚ 45.54296875 disk.rbd0.in‚0% disk./.usageGB‚1097.0261039733887 disk.dm-0.in‚4 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0$ disk.nvme0c0n1.out‚ 5342.38671875 disk.sda1.in‚0 disk.sda.out‚ 45.54296875 network.recv‚502503 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚4 disk.sda.in‚0$ memory_percent‚99.95489120483398# proc.memory.rssMB‚ 2841.06640625 disk.sda2.out‚0׋VãŽà ‚(:ƒ ›®äÔ‚ž½x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.43' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªŒ# á ‚(:þ ›®äÔ©Õz network.sent‚ 2.039974e+06) disk./.usagePercent‚73.62234511994752 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4 disk.rbd3.out‚0# proc.memory.rssMB‚ 2841.06640625 cpu‚25.26667622724865 network.recv‚508842 disk.rbd2.out‚81.5625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda.out‚45.84375 disk.rbd1.out‚ 0.67626953125 disk.nvme0n1.out‚ 5342.59375* proc.memory.percent‚17.340493202209473 disk.nvme0n1.in‚4 disk.sda2.out‚0% disk./.usageGB‚1097.0261459350586 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.dm-1.out‚45.84375$ proc.memory.availableMB‚7.296875 disk.rbd4.in‚0! disk.nvme0c0n1.out‚ 5342.59375 disk.sda3.in‚0 disk.sda3.out‚45.84375 disk.rbd1.in‚0 disk.dm-0.in‚4 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.dm-0.out‚ 5342.59375$ memory_percent‚99.95546340942383y^dÁâ ‚(:„ ª®äÔó¶Îx gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.623 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûx*¬ã ‚(:‰ ª®äÔ•’ã{ disk.rbd1.out‚ 0.67626953125 disk.dm-1.in‚0 disk.sda1.out‚0$ memory_percent‚99.95701313018799 disk.nvme0n1.in‚4 disk.dm-0.out‚ 5347.015625 disk.dm-1.out‚ 46.421875" proc.memory.rssMB‚ 2841.0703125 network.sent‚ 2.045726e+06) disk./.usagePercent‚73.62321144944339 disk.sda.in‚0 disk.sda3.out‚ 46.421875 disk.rbd2.out‚ 83.21484375 disk.rbd3.out‚0& proc.memory.availableMB‚ 7.04296875 cpu‚25.26437978059991% disk./.usageGB‚1097.0390548706055 disk.nvme0n1.out‚ 5347.015625 disk.drbd1777.out‚0 disk.rbd4.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 46.421875 disk.sda3.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.340517044067383 network.recv‚512711 disk.dm-0.in‚4 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚4 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.out‚ 5347.015625íÈ”2ä - _wandbruntime‚1310  _runtime‚1310¬cùÆå À _step‚107 & lr-AdamW/pg1‚0.0008019909940480587 & lr-AdamW/pg2‚0.0008019909940480587 trainer/global_step‚1349 $ _timestamp‚1.788417839439123e+09 _runtime‚1310.189056138Doõ$Êæ Ä & lr-AdamW/pg1‚0.0008019909940480587 & lr-AdamW/pg2‚0.0008019909940480587 trainer/global_step‚1349 $ _timestamp‚1.788417839439123e+09 _runtime‚1310.189056138 _step‚107k™tQÞ2ç - _wandbruntime‚1310  _runtime‚1310H K¨ è ¾O VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2497886097899 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2504932981760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2504932981760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2505161184256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305345536 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2497662448759 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24873771 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23643419 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53078860486123 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35121211 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55925074276352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MD`‡¹xeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚57320727220224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23642780 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 _step‚108 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281405440 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53078619797623 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14555252 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53420141294592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35095543 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 _runtime‚1310.190191596 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9087528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2554903314432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24873771 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50580974388224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50580957348864 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35095543 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59875349129216 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 trainer/global_step‚1349 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55925320481792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2505161184256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14555254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35121211 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53420159297536 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59875654474752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 $ _timestamp‚1.788417839440024e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55925074276352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59969314 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261P+2|È'é ÂO _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53420159297536 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59875654474752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 $ _timestamp‚1.788417839440024e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55925074276352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59969314 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2497886097899 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2504932981760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2504932981760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2505161184256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305345536 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2497662448759 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24873771 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23643419 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53078860486123 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35121211 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55925074276352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚57320727220224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23642780 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 _step‚108 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281405440 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53078619797623 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14555252 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53420141294592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35095543 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 _runtime‚1310.190191596 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9087528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2554903314432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24873771 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50580974388224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50580957348864 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35095543 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59875349129216 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 trainer/global_step‚1349 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55925320481792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2505161184256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14555254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35121211lË‹}²2ê - _wandbruntime‚1311  _runtime‚1311ìFÎ&ë ÈM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24891492 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24891490 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24891492 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35121094 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60038280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60012584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9093855 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2499718497707 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24891490 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14565607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35146788 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296943616 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53458497241600 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55965496133632 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2499494848359 _step‚109 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35146788 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55965249914880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14565609 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2506770676224 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59918385938432 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53116995343787 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 $ _timestamp‚1.788417840286971e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50617276846080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53116754655079 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60012584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2506770676224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35121094 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2506998892032 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23659462 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2556762025472ëþÑùì ÌM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60012584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2506770676224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35121094 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2506998892032 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23659462 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2556762025472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24891492 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24891490 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24891492 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35121094 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60038280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60012584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9093855 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2499718497707 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24891490 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14565607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35146788 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296943616 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53458497241600 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55965496133632 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2499494848359 _step‚109 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35146788 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55965249914880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14565609 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2506770676224 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59918385938432 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53116995343787 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 $ _timestamp‚1.788417840286971e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50617276846080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53116754655079m0J¨2í - _wandbruntime‚1311  _runtime‚1311BS¶Vî Ð -loss/FRt4/detection_loss‚2.40677274134424 1loss/SRqqt1/detection_loss‚2.4558078191782298 /loss/FBt1/detection_loss‚2.7700271660631355 _step‚110 1loss/FBt2/assignment_loss‚0.24891478145356416 /loss/FRt3/detection_loss‚2.3063175400289206 _runtime‚1311.038946884 2loss/SRqqt3/assignment_loss‚1.8989823600277305 0loss/SRqqt3/detection_loss‚2.167906371255716 .loss/FBt4/detection_loss‚5.330529093742371 %loss/total_loss‚55.39741024795593 epoch‚1 0loss/SRqqt2/assignment_loss‚1.69136011549463 1loss/SRqqt2/detection_loss‚1.2342115709022299 /loss/FBt2/detection_loss‚1.1707608982687816 /loss/FRt1/assignment_loss‚3.925895182291667 % _timestamp‚1.7884178402890613e+09 0loss/FRt2/assignment_loss‚2.0840206611156464 0loss/FBt1/assignment_loss‚0.5592898075950953 trainer/global_step‚1349 0loss/FRt3/assignment_loss‚3.8343351620894213 /loss/FRt4/assignment_loss‚4.796560083495246 1loss/FBt3/assignment_loss‚0.42122586940714274 1loss/FBt4/assignment_loss‚0.27770938631147146 .loss/FRt1/detection_loss‚2.940822720527649 .loss/FRt2/detection_loss‚2.139857042133808 0loss/SRqqt4/detection_loss‚2.203531695736779 .loss/FBt3/detection_loss‚4.134343425432841 2loss/SRqqt1/assignment_loss‚2.3200427599643407 2loss/SRqqt4/assignment_loss‚2.0781859940952723ÝLZï Ô 0loss/FRt2/assignment_loss‚2.0840206611156464 0loss/FBt1/assignment_loss‚0.5592898075950953 trainer/global_step‚1349 0loss/FRt3/assignment_loss‚3.8343351620894213 /loss/FRt4/assignment_loss‚4.796560083495246 1loss/FBt3/assignment_loss‚0.42122586940714274 1loss/FBt4/assignment_loss‚0.27770938631147146 .loss/FRt1/detection_loss‚2.940822720527649 .loss/FRt2/detection_loss‚2.139857042133808 0loss/SRqqt4/detection_loss‚2.203531695736779 .loss/FBt3/detection_loss‚4.134343425432841 2loss/SRqqt1/assignment_loss‚2.3200427599643407 2loss/SRqqt4/assignment_loss‚2.0781859940952723 -loss/FRt4/detection_loss‚2.40677274134424 1loss/SRqqt1/detection_loss‚2.4558078191782298 /loss/FBt1/detection_loss‚2.7700271660631355 _step‚110 1loss/FBt2/assignment_loss‚0.24891478145356416 /loss/FRt3/detection_loss‚2.3063175400289206 _runtime‚1311.038946884 2loss/SRqqt3/assignment_loss‚1.8989823600277305 0loss/SRqqt3/detection_loss‚2.167906371255716 .loss/FBt4/detection_loss‚5.330529093742371 %loss/total_loss‚55.39741024795593 epoch‚1 0loss/SRqqt2/assignment_loss‚1.69136011549463 1loss/SRqqt2/detection_loss‚1.2342115709022299 /loss/FBt2/detection_loss‚1.1707608982687816 /loss/FRt1/assignment_loss‚3.925895182291667 % _timestamp‚1.7884178402890613e+09nÈð ‚(:ƒ ¹®äԈвx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­Jñ ‚(: ¹®äÔ¡íÓz disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 0.67626953125 disk.dm-0.out‚ 5357.3515625$ memory_percent‚99.95536804199219 proc.cpu.threads‚17$ disk./.usageGB‚1097.038661956787# disk.nvme0c0n1.out‚ 5357.3515625 disk.rbd1.in‚0 disk.dm-0.in‚4 disk.sda2.out‚0 disk.dm-1.out‚ 46.87890625 disk.rbd4.in‚0 disk.sda.out‚ 46.87890625 cpu‚25.261663316109715 disk.nvme0n1.in‚4 disk.rbd0.out‚0 network.sent‚ 2.051963e+06 disk.sda3.in‚0 disk.sda3.out‚ 46.87890625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0" proc.memory.availableMB‚7.3125* proc.memory.percent‚17.340517044067383 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚4 disk.sda1.out‚0 disk.sda.in‚0) disk./.usagePercent‚73.62318508066737! disk.nvme0n1.out‚ 5357.3515625 disk.rbd4.out‚0 disk.rbd3.in‚0" proc.memory.rssMB‚ 2841.0703125 network.recv‚517644 disk.rbd2.out‚ 83.2421875.F”Ë›ò ‚(: È®äÔï·³x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚102.499' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.499799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zú#ó ‚(:˜ È®äÔÊÍöz network.recv‚523072) disk./.usagePercent‚73.62318380062969 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.67626953125% disk.nvme0n1.out‚5357.71142578125 cpu‚25.25950454774072 disk.nvme0c0n1.in‚4 disk.nvme0n1.in‚4 disk.dm-1.out‚ 47.828125 disk.rbd2.in‚0 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚5357.71142578125 disk.sda.out‚ 47.828125 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0# memory_percent‚99.9549150466919& proc.memory.availableMB‚ 7.38671875# proc.memory.rssMB‚ 2841.07421875% disk./.usageGB‚1097.0386428833008 disk.sda3.in‚0" disk.dm-0.out‚5357.71142578125 disk.rbd4.in‚0 disk.rbd2.out‚ 84.95703125 network.sent‚ 2.159344e+06 disk.rbd3.out‚0 disk.sda.in‚0 disk.dm-0.in‚4* proc.memory.percent‚17.340540885925293 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda3.out‚ 47.828125 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd4.out‚0u»³¼›ô ‚(: ×®äÔ©â¤x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.356' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.871199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡Ê(õ ‚(: ×®äÔ•®ÿ{ disk.sda.in‚0 disk.sda.out‚ 48.12109375 disk.dm-1.out‚ 48.12109375# proc.memory.rssMB‚ 2841.07421875" disk.dm-0.out‚5358.58935546875 disk.dm-1.in‚0 cpu‚25.25699926793604 disk.sda1.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚4 disk.sda2.in‚0$ memory_percent‚99.95644092559814& proc.memory.availableMB‚ 7.13671875* proc.memory.percent‚17.340540885925293 proc.cpu.threads‚17 network.recv‚527371% disk./.usageGB‚1097.1643600463867 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 48.12109375 disk.rbd1.in‚0 disk.rbd1.out‚ 0.67626953125% disk.nvme0n1.out‚5358.58935546875 disk.dm-0.in‚4( disk./.usagePercent‚73.6316207849281 disk.rbd4.in‚0 disk.nvme0c0n1.in‚4 disk.sda1.in‚0 network.sent‚ 2.165429e+06 disk.rbd2.out‚ 86.4921875 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚5358.58935546875ØbÃä2ö - _wandbruntime‚1355  _runtime‚1355dïlLÆ÷ À & lr-AdamW/pg1‚0.0008017152126802876 & lr-AdamW/pg2‚0.0008017152126802876 trainer/global_step‚1399 % _timestamp‚1.7884178845591471e+09 _runtime‚ 1355.30904696 _step‚111 Q‚eÊø Ä % _timestamp‚1.7884178845591471e+09 _runtime‚ 1355.30904696 _step‚111 & lr-AdamW/pg1‚0.0008017152126802876 & lr-AdamW/pg2‚0.0008017152126802876 trainer/global_step‚1399o=ÎN_2ù - _wandbruntime‚1355  _runtime‚1355R龜Å'ú ¿O QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57934116822016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52396080241664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57933870616576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚55337038497792 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52396097281024 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25759721 % _timestamp‚1.7884178845598514e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36402113 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36376445 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57934116822016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25759723 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚54985616950131 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15073002 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62136166 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2589296019967 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55337056500736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15073004 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59379212033024 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2596832118784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 _step‚112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚1355.310119259 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25759723 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2648057783296 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2648328702976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2597060321280 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚62027564676096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55337056500736 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57933870616576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36402113 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270919680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36376445 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2589519669107 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 trainer/global_step‚1399 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2596832118784 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55337038497792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24487570 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992DNËÉ'û ÃO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52396097281024 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25759721 % _timestamp‚1.7884178845598514e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36402113 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36376445 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57934116822016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25759723 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚54985616950131 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15073002 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62136166 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2589296019967 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55337056500736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15073004 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59379212033024 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2596832118784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 _step‚112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚1355.310119259 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25759723 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2648057783296 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2648328702976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2597060321280 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚62027564676096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55337056500736 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57933870616576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36402113 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270919680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36376445 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2589519669107 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 trainer/global_step‚1399 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2596832118784 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55337038497792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24487570 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57934116822016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52396080241664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57933870616576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚55337038497792 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256pÁ¤Øo%ü - _wandbruntime‚1356  _r&]åÚ untime‚1356@ßcÿÏ&ý ÉM BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36427690 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24503346 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 trainer/global_step‚1399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55023751807795 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36401996 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15083359 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52432382699520 % _timestamp‚1.7884178854177835e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268809216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2591128419567 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25777440 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62205132 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25777442 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2598669813248 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57974292473856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2649928125440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 _runtime‚1356.168392227 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24503978 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36427690 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292749312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52432399738880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15083357 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55375394444800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62179436 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25777440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚59420381729280 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55023511119087 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55375394444800 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55375376441856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2598898029056 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2650196934656 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57974046255104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62179436 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25777442 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55375394444800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62179436 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25777440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚59420381729280 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55023511119087 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55375394444800 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55375376441856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2598898029056 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2650196934656 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57974046255104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62179436 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25777442 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36427690 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24503346 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 trainer/global_step‚1399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55023751807795 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36401996 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15083359 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52432382699520 % _timestamp‚1.7884178854177835e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268809216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2591128419567 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25777440 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62205132 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25777442 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2598669813248 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57974292473856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2649928125440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 _runtime‚1356.168392227 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24503978 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36427690 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292749312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52432399738880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561q¥<Üü2ÿ - _wandbruntime‚1356  _runtime‚1356.ª)T€Î 2loss/SRqqt3/assignment_loss‚2.2510007716888607 /loss/FBt1/detection_loss‚2.5521938800811768 .loss/FRt2/detection_loss‚1.982730679710706 /loss/FRt3/assignment_loss‚4.127130222320557 0loss/FBt3/assignment_loss‚0.7374852870901426 _runtime‚1356.169531212 /loss/FRt2/assignment_loss‚2.724194476339552 /loss/FRt1/detection_loss‚3.0634064907208085 1loss/SRqqt4/detection_loss‚2.1293040477648013 /loss/FBt3/detection_loss‚3.9776291052500405 $loss/total_loss‚54.7663381396462 /loss/FBt2/detection_loss‚0.6553542927916473 /loss/FRt1/assignment_loss‚4.422213658690453 2loss/SRqqt4/assignment_loss‚2.2571637708602883 .loss/FRt3/detection_loss‚2.829418006965092 _step‚114 /loss/FRt4/assignment_loss‚4.762651298727308 1loss/SRqqt1/detection_loss‚2.5854222223614203 % _timestamp‚1.7884178854197202e+09 1loss/SRqqt1/assignment_loss‚2.117295795317852 epoch‚1 trainer/global_step‚1399 0loss/FBt2/assignment_loss‚0.2539001246577414 0loss/SRqqt2/detection_loss‚1.020228695386732 /loss/FBt4/detection_loss‚3.1727690279483793 2loss/SRqqt2/assignment_loss‚1.7565929580112578 0loss/FBt1/assignment_loss‚0.6962362885242328 0loss/FBt4/assignment_loss‚0.3986001625657082 .loss/FRt4/detection_loss‚2.333946114494687 0loss/SRqqt3/detection_loss‚1.959470761376758ÙZNëXÒ 1loss/SRqqt1/assignment_loss‚2.117295795317852 epoch‚1 trainer/global_step‚1399 0loss/FBt2/assignment_loss‚0.2539001246577414 0loss/SRqqt2/detection_loss‚1.020228695386732 /loss/FBt4/detection_loss‚3.1727690279483793 2loss/SRqqt2/assignment_loss‚1.7565929580112578 0loss/FBt1/assignment_loss‚0.6962362885242328 0loss/FBt4/assignment_loss‚0.3986001625657082 .loss/FRt4/detection_loss‚2.333946114494687 0loss/SRqqt3/detection_loss‚1.959470761376758 2loss/SRqqt3/assignment_loss‚2.2510007716888607 /loss/FBt1/detection_loss‚2.5521938800811768 .loss/FRt2/detection_loss‚1.982730679710706 /loss/FRt3/assignment_loss‚4.127130222320557 0loss/FBt3/assignment_loss‚0.7374852870901426 _runtime‚1356.169531212 /loss/FRt2/assignment_loss‚2.724194476339552 /loss/FRt1/detection_loss‚3.0634064907208085 1loss/SRqqt4/detection_loss‚2.1293040477648013 /loss/FBt3/detection_loss‚3.9776291052500405 $loss/total_loss‚54.7663381396462 /loss/FBt2/detection_loss‚0.6553542927916473 /loss/FRt1/assignment_loss‚4.422213658690453 2loss/SRqqt4/assignment_loss‚2.2571637708602883 .loss/FRt3/detection_loss‚2.829418006965092 _step‚114 /loss/FRt4/assignment_loss‚4.762651298727308 1loss/SRqqt1/detection_loss‚2.5854222223614203 % _timestamp‚1.7884178854197202e+09r=¹'›‚‚(: æ®äԊΦx gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.436' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.887200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,Æî Aƒ‚(:¶ æ®äÔ‹ÞØ{$ memory_percent‚99.95753765106201% disk./.usageGB‚1097.1633224487305% disk.nvme0n1.out‚5457.91357421875" disk.dm-0.out‚5457.91357421875 disk.drbd1777.out‚0 disk.sda.out‚ 55.85546875 disk.sda3.out‚ 55.85546875# proc.memory.rssMB‚ 2841.07421875 disk.sda1.in‚0 network.sent‚ 2.171174e+06 disk.sda2.in‚0 disk.dm-1.out‚ 55.85546875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚5457.91357421875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd2.out‚ 86.5703125 disk.rbd3.in‚0 disk.sda.in‚ 0.8203125& proc.memory.availableMB‚ 6.95703125 network.recv‚531425 disk.rbd1.out‚ 0.67626953125 disk.dm-1.in‚ 0.8203125 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.8203125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4* proc.memory.percent‚17.340540885925293 cpu‚25.25477930945479 disk.nvme0n1.in‚4 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.dm-0.in‚4) disk./.usagePercent‚73.63155115087879 disk.drbd1777.in‚0 disk.rbd3.out‚0]~Ì Ž„‚(:ƒ õ®äÔÑæ°x gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü2çh…‚(:Ý õ®äÔâ‡åz) disk./.usagePercent‚73.63997098267244 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0' disk.nvme0c0n1.out‚5561.73974609375 disk.sda3.in‚4.0625 disk.sda3.out‚ 75.796875 disk.rbd1.in‚0 disk.dm-0.in‚ 4.9296875 network.sent‚ 2.279906e+06 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd3.in‚0 disk.sda2.out‚ 0.0390625 cpu‚25.252450923044176 proc.cpu.threads‚17# proc.memory.rssMB‚ 2841.07421875 network.recv‚538399 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 75.8359375" disk.dm-0.out‚5561.73974609375 disk.sda.in‚ 4.11328125 disk.nvme0n1.in‚ 4.9296875% proc.memory.availableMB‚ 6.7421875 disk.dm-1.in‚4.0625 disk.dm-1.out‚ 75.796875% disk.nvme0n1.out‚5561.73974609375 disk.sda2.in‚0* proc.memory.percent‚17.340540885925293% disk./.usageGB‚1097.2887840270996 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 0.67626953125$ memory_percent‚99.95884895324707 disk.rbd2.out‚ 88.109375 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125ÚÊa›†‚(: „¯äԙόx gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.436' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.887200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê©ïi‡‚(:Þ „¯äÔþÏÓ{ disk.nvme0n1.in‚ 4.9296875 disk.dm-1.out‚ 86.921875' disk.nvme0c0n1.out‚5661.21533203125 disk.sda.out‚ 86.9609375$ memory_percent‚99.95932579040527 disk.dm-1.in‚6.328125 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda3.in‚6.328125) disk./.usagePercent‚73.65674894844393% disk./.usageGB‚1097.5387878417969 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 0.67626953125% disk.nvme0n1.out‚5661.21533203125 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd3.in‚0# proc.memory.rssMB‚ 2841.07421875* proc.memory.percent‚17.340540885925293 disk.rbd2.out‚88.3125" disk.dm-0.out‚5661.21533203125 disk.rbd0.out‚0 disk.sda.in‚ 6.37890625 disk.rbd1.in‚0 disk.rbd3.out‚0 network.sent‚ 2.285021e+06 disk.sda1.out‚0% proc.memory.availableMB‚ 6.6640625 cpu‚25.25028331639407 network.recv‚541265 disk.rbd4.out‚0 disk.sda3.out‚ 86.921875 proc.cpu.threads‚17Ž²; 2ˆ- _wandbruntime‚1400  _runtime‚1400{±@(ljÁ _step‚115 & lr-AdamW/pg1‚0.0008013559560680854 & lr-AdamW/pg2‚0.0008013559560680854 trainer/global_step‚1449 % _timestamp‚1.7884179299381406e+09 _runtime‚1400.6880744061&ÒOËŠÅ & lr-AdamW/pg2‚0.0008013559560680854 trainer/global_step‚1449 % _timestamp‚1.7884179299381406e+09 _runtime‚1400.688074406 _step‚115 & lr-AdamW/pg1‚0.0008013559560680854súÅ¢j2‹- _wandbruntime‚1400  _runtime‚1400¤ŸHÂ0Œ¾O ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2688959458304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277211136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1449 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61437720785920 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59942913162240 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2688959458304 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61437696845824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2681153240315 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37683015 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26645673 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚647 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2741806907904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54211220173824 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64179226542592 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59942666956800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166!½óÿÈ'ÂO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37683015 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26645673 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚647 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2741806907904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54211220173824 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64179226542592 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59942666956800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2688959458304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277211136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1449 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61437720785920 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59942913162240 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2688959458304 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61437696845824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2681153240315t¶Wi§2Ž- _wandbruntime‚1401  _runtime‚1401ÒFífÏ&ÉM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301137920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _runtime‚1401.553071732 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64346288 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2690797166080 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37708592 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9739169 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57292273645056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15601109 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2682985640123 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37682898 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26663392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2743668832256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2690797166080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59983088814080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64222559314432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 trainer/global_step‚1449 _step‚117 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25339659 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56930508271803 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2682761990775 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59983088814080 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59982842595328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26663390 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64371984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25340278 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54247505592320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2690568950272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26663392 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37682898 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619ÙßyH€ÍM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37708592 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9739169 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57292273645056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15601109 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2682985640123 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37682898 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26663392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2743668832256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2690797166080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59983088814080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64222559314432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 trainer/global_step‚1449 _step‚117 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25339659 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56930508271803 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2682761990775 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59983088814080 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59982842595328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26663390 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64371984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25340278 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54247505592320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2690568950272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26663392 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37682898 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301137920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _runtime‚1401.553071732 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64346288 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2690797166080 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0u.'[„2‘- _wandbruntime‚1401  _runtime‚1401ò0W’Ñ 2loss/SRqqt2/assignment_loss‚1.8717545236532505 0loss/FBt1/assignment_loss‚0.7499191345026096 0loss/FBt2/assignment_loss‚0.2294593970923781 1loss/SRqqt3/detection_loss‚1.7523242926822518 epoch‚1 /loss/FRt1/assignment_loss‚4.540694082105482 /loss/FBt4/assignment_loss‚0.921149742603302 /loss/FBt2/detection_loss‚0.7203169022406261 trainer/global_step‚1449 % _timestamp‚1.7884179308042896e+09 /loss/FRt2/assignment_loss‚3.098235951817554 2loss/SRqqt1/assignment_loss‚2.4846441660608565 0loss/SRqqt2/detection_loss‚1.241306950381169 1loss/SRqqt4/detection_loss‚2.0511891753115554 /loss/FBt4/detection_loss‚3.2145609259605408 &loss/total_loss‚55.528900284997775 _step‚118 0loss/FBt3/assignment_loss‚0.3283674865909916 .loss/FRt1/detection_loss‚2.776192147989531 /loss/FRt3/detection_loss‚2.3568131777704977 /loss/FRt4/detection_loss‚2.6211795773771076 0loss/SRqqt1/detection_loss‚2.658550703525543 2loss/SRqqt4/assignment_loss‚2.2065588730446835 _runtime‚1401.554102674 /loss/FRt4/assignment_loss‚4.480947772661845 /loss/FBt1/detection_loss‚2.2709353466828666 2loss/SRqqt3/assignment_loss‚1.7878788330082624 /loss/FRt2/detection_loss‚2.2055983708604523 .loss/FBt3/detection_loss‚4.739644110202789 /loss/FRt3/assignment_loss‚4.220678640871632Ðw.[“Õ 2loss/SRqqt4/assignment_loss‚2.2065588730446835 _runtime‚1401.554102674 /loss/FRt4/assignment_loss‚4.480947772661845 /loss/FBt1/detection_loss‚2.2709353466828666 2loss/SRqqt3/assignment_loss‚1.7878788330082624 /loss/FRt2/detection_loss‚2.2055983708604523 .loss/FBt3/detection_loss‚4.739644110202789 /loss/FRt3/assignment_loss‚4.220678640871632 2loss/SRqqt2/assignment_loss‚1.8717545236532505 0loss/FBt1/assignment_loss‚0.7499191345026096 0loss/FBt2/assignment_loss‚0.2294593970923781 1loss/SRqqt3/detection_loss‚1.7523242926822518 epoch‚1 /loss/FRt1/assignment_loss‚4.540694082105482 /loss/FBt4/assignment_loss‚0.921149742603302 /loss/FBt2/detection_loss‚0.7203169022406261 trainer/global_step‚1449 % _timestamp‚1.7884179308042896e+09 /loss/FRt2/assignment_loss‚3.098235951817554 2loss/SRqqt1/assignment_loss‚2.4846441660608565 0loss/SRqqt2/detection_loss‚1.241306950381169 1loss/SRqqt4/detection_loss‚2.0511891753115554 /loss/FBt4/detection_loss‚3.2145609259605408 &loss/total_loss‚55.528900284997775 _step‚118 0loss/FBt3/assignment_loss‚0.3283674865909916 .loss/FRt1/detection_loss‚2.776192147989531 /loss/FRt3/detection_loss‚2.3568131777704977 /loss/FRt4/detection_loss‚2.6211795773771076 0loss/SRqqt1/detection_loss‚2.658550703525543v… \¡›”‚(: “¯äԮؠx gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.568' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.113599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×æKo•‚(:ä “¯äÔ¢œáz proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 89.84765625( disk./.usagePercent‚73.6567502284816 disk.rbd4.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2841.07421875 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda.in‚ 7.23828125 disk.rbd1.out‚ 0.67626953125 network.sent‚ 2.29029e+06 network.recv‚545524 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0' disk.nvme0c0n1.out‚5670.69482421875 disk.sda.out‚ 146.51953125% disk./.usageGB‚1097.5388069152832 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.9296875$ memory_percent‚99.95980262756348 disk.rbd0.out‚0" disk.dm-0.out‚5670.69482421875 disk.sda2.in‚0 disk.dm-1.out‚ 146.48046875 disk.sda3.in‚7.1875 disk.sda3.out‚ 146.48046875 disk.rbd1.in‚0% disk.nvme0n1.out‚5670.69482421875 disk.dm-1.in‚7.1875 disk.sda1.in‚ 0.05078125* proc.memory.percent‚17.340540885925293 cpu‚25.24822415091348% proc.memory.availableMB‚ 6.5859375àî¿éš–‚(: ¢¯äÔ£†™x gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚93.105' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.621000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&õËh—‚(:Ý ¢¯äÔ²¿è{ disk.rbd1.out‚ 0.67626953125 disk.dm-1.out‚ 147.06640625 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.23828125* proc.memory.percent‚17.340564727783203% disk./.usageGB‚1097.5387878417969 disk.sda2.in‚0 disk.dm-1.in‚7.1875 disk.rbd3.in‚0 disk.sda3.out‚ 147.06640625 network.recv‚551162 disk.rbd1.in‚0# disk.nvme0n1.out‚5769.904296875 disk.sda1.in‚ 0.05078125 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625 disk.sda.out‚ 147.10546875$ proc.memory.availableMB‚6.328125 cpu‚25.24613759050987 disk.nvme0c0n1.in‚ 4.9296875 disk.nvme0n1.in‚ 4.9296875 disk.dm-0.out‚5769.904296875 disk.rbd2.in‚0! proc.memory.rssMB‚ 2841.078125 network.sent‚ 2.398361e+06 disk.dm-0.in‚ 4.9296875 disk.rbd2.out‚ 89.92578125 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0$ memory_percent‚99.96137619018555) disk./.usagePercent‚73.65674894844393 disk.drbd1777.in‚ 1.03515625% disk.nvme0c0n1.out‚5769.904296875 disk.sda3.in‚7.1875*OE›˜‚(: ±¯äÔ×ÇÍx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.138' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.027600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËÌLÉk™‚(:à ±¯äÔü‘¿{( disk./.usagePercent‚73.6567502284816$ disk.nvme0n1.out‚5869.3154296875& proc.memory.availableMB‚ 6.33203125 network.recv‚556127 disk.sda.out‚ 147.67578125& disk.nvme0c0n1.out‚5869.3154296875! proc.memory.rssMB‚ 2841.078125 cpu‚25.24414300463829 disk.rbd3.out‚0 disk.dm-0.in‚ 4.9296875 disk.rbd2.in‚0 disk.rbd4.out‚0 network.sent‚ 2.40467e+06 disk.rbd1.in‚0$ memory_percent‚99.96135234832764 disk.rbd4.in‚0% disk./.usageGB‚1097.5388069152832 disk.rbd3.in‚0 disk.sda.in‚ 7.23828125 disk.sda3.in‚7.1875 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 147.63671875 disk.nvme0n1.in‚ 4.9296875 disk.dm-1.in‚7.1875 disk.drbd1777.in‚ 1.03515625* proc.memory.percent‚17.340564727783203! disk.dm-0.out‚5869.3154296875 disk.sda2.in‚0 disk.dm-1.out‚ 147.63671875 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.in‚ 0.05078125 proc.cpu.threads‚17 disk.rbd1.out‚ 0.67626953125 disk.rbd2.out‚ 91.46484375 disk.rbd0.in‚0 disk.rbd0.out‚0»Ó0P2š- _wandbruntime‚1445  _runtime‚1445­ANÆ›À & lr-AdamW/pg1‚0.0008009132990799917 & lr-AdamW/pg2‚0.0008009132990799917 trainer/global_step‚1499 % _timestamp‚1.7884179751444254e+09 _runtime‚ 1445.89437186 _step‚119OþŠ­ÊœÄ & lr-AdamW/pg1‚0.0008009132990799917 & lr-AdamW/pg2‚0.0008009132990799917 trainer/global_step‚1499 % _timestamp‚1.7884179751444254e+09 _runtime‚ 1445.89437186 _step‚119w@ZJ2- _wandbruntime‚1445  _runtime‚1445ÞoûØÅ'ž¿O JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66495540 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2835034560000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63496205598720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38963917 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38938249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10056569 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56026343066624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2772786811523 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281405440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 trainer/global_step‚1499 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10057241 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚66331521564160 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚66331216218624 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58799129878147 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _step‚120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2780858595328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66469870 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16108504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61951463297024 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 % _timestamp‚1.7884179751451774e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61951709502464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2780630392832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140¦<º‰É'ŸÃO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2835034560000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63496205598720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38963917 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38938249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10056569 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56026343066624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2772786811523 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281405440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 trainer/global_step‚1499 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10057241 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚66331521564160 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚66331216218624 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58799129878147 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _step‚120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2780858595328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66469870 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16108504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61951463297024 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 % _timestamp‚1.7884179751451774e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61951709502464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2780630392832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66495540 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262xt˜p2 - _wandbruntime‚1446  _runtime‚1446TdÇ:…¡ËM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38989494 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38963800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61991638935552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16118857 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292749312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2774395561983 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66538836 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66513140 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2782468087296 VHDeviceStatsMonitor.on_train_batcPháL h_end/allocated_bytes.small_pool.current‚ 228215808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38963800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59209170848256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59209170848256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10062940 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66513140 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26182472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16118859 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1499 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61991885154304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59209188851200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26181797 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2837176448000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚1446.744195916 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59209188851200 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63537351354880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10063613 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 % _timestamp‚1.7884179759934573e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2782696303104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38989494 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2836907638784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27549340 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58837264735811 _step‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58837024047103 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2774619211331 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268809216 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2782696303104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚66374551742976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27549342 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27549342 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2782468087296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56062645524480 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27549340 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚66374258993664 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56062628485120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58837024047103 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2774619211331 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268809216 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2782696303104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚66374551742976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27549342 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27549342 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2782468087296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56062645524480 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27549340 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚66374258993664 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56062628485120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66538836 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66513140 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2782468087296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38963800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59209170848256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59209170848256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10062940 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66513140 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26182472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16118859 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1499 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61991885154304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59209188851200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26181797 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2837176448000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚1446.744195916 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59209188851200 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63537351354880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10063613 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 % _timestamp‚1.7884179759934573e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2782696303104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38989494 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2836907638784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27549340 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58837264735811 _step‚121yïž:2£- _wandbruntime‚1446  _runtime‚1446A`JS¤Í % _timestamp‚1.7884179759953947e+09 .loss/FRt4/detection_loss‚2.482672861436518 /loss/FBt3/detection_loss‚2.8578542470932007 trainer/global_step‚1499 _runtime‚1446.745223782 _step‚122 /loss/FRt4/assignment_loss‚4.677833086106835 1loss/SRqqt3/assignment_loss‚2.340471183800179 .loss/FRt1/detection_loss‚2.695323845744133 .loss/FRt3/detection_loss‚2.437468113852482 2loss/SRqqt2/assignment_loss‚1.5676642652778399 /loss/FBt3/assignment_loss‚0.702944241464138 epoch‚1 .loss/FRt1/assignment_loss‚5.07745490372181 /loss/FBt2/detection_loss‚0.7231459428452783 2loss/SRqqt4/assignment_loss‚1.9992054215803439 0loss/FBt2/assignment_loss‚0.2525762171500419 1loss/FBt4/assignment_loss‚0.49037283658981323 /loss/FRt2/detection_loss‚2.1633649406333766 0loss/FBt1/assignment_loss‚0.9052584996590247 0loss/SRqqt1/detection_loss‚2.440646594390273 1loss/SRqqt3/detection_loss‚2.1033756538577704 0loss/SRqqt1/assignment_loss‚2.54508303059265 1loss/SRqqt2/detection_loss‚1.1944531458947394 .loss/FBt1/detection_loss‚2.229406067958245 %loss/total_loss‚53.95217496566631 0loss/FRt2/assignment_loss‚2.1589979579051337 /loss/FRt3/assignment_loss‚4.034048978020163 1loss/SRqqt4/detection_loss‚1.9797706591839692 /loss/FBt4/detection_loss‚3.8927822709083557ÿ®W¥Ñ .loss/FRt4/detection_loss‚2.482672861436518 /loss/FBt3/detection_loss‚2.8578542470932007 trainer/global_step‚1499 _runtime‚1446.745223782 _step‚122 /loss/FRt4/assignment_loss‚4.677833086106835 1loss/SRqqt3/assignment_loss‚2.340471183800179 .loss/FRt1/detection_loss‚2.695323845744133 .loss/FRt3/detection_loss‚2.437468113852482 2loss/SRqqt2/assignment_loss‚1.5676642652778399 /loss/FBt3/assignment_loss‚0.702944241464138 epoch‚1 .loss/FRt1/assignment_loss‚5.07745490372181 /loss/FBt2/detection_loss‚0.7231459428452783 2loss/SRqqt4/assignment_loss‚1.9992054215803439 0loss/FBt2/assignment_loss‚0.2525762171500419 1loss/FBt4/assignment_loss‚0.49037283658981323 /loss/FRt2/detection_loss‚2.1633649406333766 0loss/FBt1/assignment_loss‚0.9052584996590247 0loss/SRqqt1/detection_loss‚2.440646594390273 1loss/SRqqt3/detection_loss‚2.1033756538577704 0loss/SRqqt1/assignment_loss‚2.54508303059265 1loss/SRqqt2/detection_loss‚1.1944531458947394 .loss/FBt1/detection_loss‚2.229406067958245 %loss/total_loss‚53.95217496566631 0loss/FRt2/assignment_loss‚2.1589979579051337 /loss/FRt3/assignment_loss‚4.034048978020163 1loss/SRqqt4/detection_loss‚1.9797706591839692 /loss/FBt4/detection_loss‚3.8927822709083557 % _timestamp‚1.7884179759953947e+09z%éSŽ¦‚(:ƒ À¯äÔÊíªx gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚101.01' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢'šIg§‚(:Ü À¯äÔÆüîz disk.sda3.out‚ 148.171875$ disk.nvme0n1.out‚5968.4677734375 disk.sda2.in‚0 disk.rbd2.out‚ 91.54296875 disk.rbd0.out‚0$ proc.memory.availableMB‚6.515625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9296875& disk.nvme0c0n1.out‚5968.4677734375 disk.rbd1.in‚0 disk.rbd0.in‚0! disk.dm-0.out‚5968.4677734375$ memory_percent‚99.96023178100586% disk./.usageGB‚1098.0388107299805 disk.sda1.in‚ 0.05078125 disk.dm-1.in‚7.1875 network.recv‚559762 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.sda.in‚ 7.23828125 disk.rbd1.out‚ 0.67626953125 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd4.out‚0 disk.sda.out‚ 148.2109375 disk.sda3.in‚7.1875 disk.dm-0.in‚ 4.9296875! proc.memory.rssMB‚ 2841.078125 cpu‚25.242030368316865 network.sent‚ 2.409994e+06) disk./.usagePercent‚73.69030590401708 disk.rbd3.out‚0 disk.nvme0n1.in‚ 4.9296875 disk.dm-1.out‚ 148.171875* proc.memory.percent‚17.3405647277832036é•5Ž¨‚(:ƒ ϯäÔ×…«x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.68' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™7…/a©‚(:Ö ϯäÔÊåï{ network.sent‚ 2.518364e+06 disk.rbd3.out‚0 disk.rbd1.out‚ 0.67626953125 disk.nvme0n1.in‚ 4.9296875 disk.dm-0.out‚5968.728515625 network.recv‚566279 disk.dm-1.out‚148.9375 disk.rbd2.in‚0% disk.nvme0c0n1.out‚5968.728515625 disk.rbd3.in‚0 disk.sda3.out‚148.9375 disk.rbd1.in‚0$ memory_percent‚99.96025562286377) disk./.usagePercent‚73.69030872009996% disk./.usageGB‚1098.0388526916504 disk.rbd4.out‚0 disk.sda.in‚ 7.23828125 disk.sda2.out‚ 0.0390625& proc.memory.availableMB‚ 6.51171875* proc.memory.percent‚17.340564727783203 disk.rbd4.in‚0 disk.rbd2.out‚ 91.5546875 disk.rbd0.out‚0# disk.nvme0n1.out‚5968.728515625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda2.in‚0 cpu‚25.240070432396802 disk.dm-1.in‚7.1875 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.1875 disk.dm-0.in‚ 4.9296875! proc.memory.rssMB‚ 2841.078125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.out‚ 148.9765625(€Ž}‹ª‚(:€ Þ¯äÔËø™x gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.5' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ajMØ^«‚(:Ó Þ¯äÔˆ°Àz disk.dm-0.out‚6067.873046875 disk.nvme0c0n1.in‚ 4.9296875 disk.sda.in‚ 7.23828125* proc.memory.percent‚17.340564727783203 disk.dm-1.in‚7.1875 disk.sda.out‚ 149.5390625 disk.sda3.in‚7.1875 disk.sda3.out‚149.5$ memory_percent‚99.96025562286377 cpu‚25.23806614578294% disk./.usageGB‚1098.0380363464355 disk.rbd2.out‚91.65673828125 disk.rbd0.out‚0) disk./.usagePercent‚73.69025393448763 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚149.5 disk.sda1.out‚0 disk.rbd3.out‚0& proc.memory.availableMB‚ 6.51171875 network.sent‚ 2.524282e+06 disk.sda2.in‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 network.recv‚570354 disk.nvme0n1.in‚ 4.9296875 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.67626953125# disk.nvme0n1.out‚6067.873046875% disk.nvme0c0n1.out‚6067.873046875! proc.memory.rssMB‚ 2841.078125 proc.cpu.threads‚17 disk.dm-0.in‚ 4.9296875 disk.rbd2.in‚0 disk.rbd3.in‚0Ë€Lã2¬- _wandbruntime‚1491  _runtime‚1491—/CTÇ­Á & lr-AdamW/pg1‚0.0008003873339650596 & lr-AdamW/pg2‚0.0008003873339650596 trainer/global_step‚1549 % _timestamp‚1.7884180204227133e+09 _runtime‚1491.172704267 _step‚123QÓ !ˮŠ& lr-AdamW/pg1‚0.0008003873339650596 & lr-AdamW/pg2‚0.0008003873339650596 trainer/global_step‚1549 % _timestamp‚1.7884180204227133e+09 _runtime‚1491.172704267 _step‚123{¿÷Õ©2¯- _wandbruntime‚1491  _runtime‚1491ÌôÑ'аÁO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68662392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚57841465959424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚57841448920064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27006845 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28417571 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10381249 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28417573 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16626252 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60705886342155 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40244819 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61087730107392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10380593 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2928737288704 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40219151 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚63960259637248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60705645653655 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61087730107392 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚68483126548992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40244819 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2928460077568 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63960505842688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28417571 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277211136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2872757732352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16626254 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884180204235034e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63960259637248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872ad^èË'±ÅO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2928460077568 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63960505842688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28417571 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277211136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2872757732352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16626254 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884180204235034e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63960259637248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68662392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚57841465959424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚57841448920064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27006845 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28417571 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10381249 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28417573 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16626252 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60705886342155 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40244819 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61087730107392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10380593 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2928737288704 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40219151 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚63960259637248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60705645653655 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61087730107392 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚68483126548992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40244819|0Æy2²- _wandbruntime‚1492  _runtime‚1492`ØæÐ&³ÊM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10387126 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28435292 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57877768417280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60744021199819 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16636607 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2866252782539 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64000681494528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68705688 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2874595440128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16636609 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2874595440128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68526162115072 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40244702 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61126068051456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40270396 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27023733 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10387793 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2866029133191 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2874367224320 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68679992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27024402 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 1492.03220061 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64000435275776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚57877751377920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884180212817638e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40270396 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68705688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096Qå@Ô#´ÎM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28435290 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68705688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10387126 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28435292 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57877768417280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60744021199819 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16636607 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2866252782539 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64000681494528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68705688 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2874595440128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16636609 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2874595440128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68526162115072 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40244702 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61126068051456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40270396 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27023733 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10387793 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2866029133191 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2874367224320 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68679992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27024402 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 1492.03220061 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64000435275776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚57877751377920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884180212817638e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40270396 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266725376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2955830304799 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29303521 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17144002 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2964656869376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2964428666880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41525721 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65969302182912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2964428666880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67613175224320 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70635280496128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65969055977472 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41500053 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65969055977472 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63004645313536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2964656869376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚62612642806163 _runtime‚1536.201265373 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70634989830656 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚62612402117663 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 trainer/global_step‚1599 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70829244 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59656571812864 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63004645313536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0×fÙ\Ì'ÃÆO _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59656588852224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266725376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2955830304799 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29303521 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17144002 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2964656869376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2964428666880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41525721 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65969302182912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2964428666880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67613175224320 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70635280496128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65969055977472 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41500053 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65969055977472 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63004645313536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2964656869376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚62612642806163 _runtime‚1536.201265373 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70634989830656 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚62612402117663 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 trainer/global_step‚1599 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70829244 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59656571812864 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63004645313536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67613151284224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70803574 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41525721 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚655 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 % _timestamp‚1.7884180654511173e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29303523 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63004627310592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10705247 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3022105271808€Ïþºç2Ä- _wandbruntime‚1537  _runtime‚1537[¤Þ@ ÅÊM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67654320980480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62650777663827 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 _step‚129 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67654344920576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41551298 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29321240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41551298 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚63042965254656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10712093 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3023977370624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70872540 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29321242 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70678029541888 BDeviceStatsMonitor.on_train_batch_end/allocated_bytesî¤u™.all.peak‚ 6739343872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29321240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70846844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17154359 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2957886353747 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27866452 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10711426 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29321242 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2957662704399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2966266361344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41525604 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70678322291200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63042983257600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70846844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27865783 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 $ _timestamp‚1.788418066307172e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚63042965254656 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70872540 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292749312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59692874270720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59692891310080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚1537.057968628 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ˆÙîzÕ&ÆÏM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚66009477834752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41525604 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3023708561408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268809216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚63042983257600 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2966494577152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66009477834752 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70872540 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292749312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59692874270720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59692891310080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚1537.057968628 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67654320980480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62650777663827 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 _step‚129 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67654344920576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41551298 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29321240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41551298 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚63042965254656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10712093 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3023977370624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70872540 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29321242 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70678029541888 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29321240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70846844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17154359 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2957886353747 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27866452 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10711426 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29321242 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2957662704399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2966266361344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41525604 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70678322291200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63042983257600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70846844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27865783 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 $ _timestamp‚1.788418066307172e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚63042965254656 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11031625 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273016832 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67977852317696 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64921524513792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3056556006400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67978098523136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42806623 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64921524513792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11032261 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚61471694705664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30189473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67978098523136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42780955 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72786930831872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1581.362476385 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3056327803904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64921542516736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67977852317696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64519158581671"ïÜÌ'ÕÆO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42806623 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28694015 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3115567751680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72787227788800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11031625 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273016832 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67977852317696 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64921524513792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3056556006400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67978098523136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42806623 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64921524513792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11032261 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚61471694705664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30189473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67978098523136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42780955 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72786930831872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1581.362476385 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3056327803904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64921542516736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67977852317696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64519158581671 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72970426 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299„ <\2Ö- _wandbruntime‚1582  _runtime‚1582"aWeÐ&×ÊM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30207190 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68018274174976 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 _runtime‚ 1582.22629127 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3058165498368 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42832200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42806506 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3058165498368 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68018027956224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3117159219200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275100672 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61507997163520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73013696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28709958 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚1649 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28710605 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72830264053248 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3058393714176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30207192 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72829965012480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68018274174976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64959880460800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11038496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64557534127835 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73013696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73039392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299040768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚69712829733376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚69712805793280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64959862457856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17672107 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30207192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64557293439127 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64959880460800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42806506 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3049519924955 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64959862457856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3058393714176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17672109 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚61508014202880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚133 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73039392 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚61508014202880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚133 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73039392 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3117159219200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275100672 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61507997163520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73013696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28709958 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚1649 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28710605 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72830264053248 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3058393714176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30207192 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72829965012480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68018274174976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64959880460800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11038496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64557534127835 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73013696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73039392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299040768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚69712829733376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚69712805793280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64959862457856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17672107 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30207192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64557293439127 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64959880460800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42806506 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3049519924955 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348…þ ±ç2Ù- _wandbruntime‚1582  _runtime‚1582u&¦~ÚÌ 0loss/FBt1/assignment_loss‚0.5722330956741644 0loss/FBt4/assignment_loss‚0.5184135586023331 0loss/SRqqt2/detection_loss‚1.072918736158985 _step‚134 /loss/FRt3/assignment_loss‚4.449944891035557 2loss/SRqqt3/assignment_loss‚2.0572330361163176 0loss/FBt3/assignment_loss‚0.2515817905465762 /loss/FRt3/detection_loss‚2.4486827502648034 1loss/SRqqt1/detection_loss‚2.7715255093574522 2loss/SRqqt4/assignment_loss‚2.4482659293072566 /loss/FRt1/detection_loss‚2.5030928097273173 /loss/FRt2/detection_loss‚1.9592313862272672 _runtime‚ 1582.22734726 2loss/SRqqt1/assignment_loss‚2.397XfÔ28681778907777 /loss/FBt2/assignment_loss‚0.234141495883988 /loss/FBt1/detection_loss‚3.4312641248106956 /loss/FBt2/detection_loss‚0.8050789222093847 1loss/SRqqt3/detection_loss‚1.8375253296560712 /loss/SRqqt4/detection_loss‚2.03121838505779 /loss/FRt1/assignment_loss‚4.185014825118215 2loss/SRqqt2/assignment_loss‚1.7083660011524207 .loss/FRt4/detection_loss‚2.529040356477102 /loss/FBt4/detection_loss‚3.7547284364700317 %loss/total_loss‚55.02374803726308 $ _timestamp‚1.788418111477546e+09 .loss/FRt2/assignment_loss‚2.56710892702852 .loss/FBt3/detection_loss‚3.702256759007772 /loss/FRt4/assignment_loss‚4.792012803482287 epoch‚1 trainer/global_step‚1649`+zcWÛÑ .loss/FBt3/detection_loss‚3.702256759007772 /loss/FRt4/assignment_loss‚4.792012803482287 epoch‚1 trainer/global_step‚1649 0loss/FBt1/assignment_loss‚0.5722330956741644 0loss/FBt4/assignment_loss‚0.5184135586023331 0loss/SRqqt2/detection_loss‚1.072918736158985 _step‚134 /loss/FRt3/assignment_loss‚4.449944891035557 2loss/SRqqt3/assignment_loss‚2.0572330361163176 0loss/FBt3/assignment_loss‚0.2515817905465762 /loss/FRt3/detection_loss‚2.4486827502648034 1loss/SRqqt1/detection_loss‚2.7715255093574522 2loss/SRqqt4/assignment_loss‚2.4482659293072566 /loss/FRt1/detection_loss‚2.5030928097273173 /loss/FRt2/detection_loss‚1.9592313862272672 _runtime‚ 1582.22734726 2loss/SRqqt1/assignment_loss‚2.3928681778907777 /loss/FBt2/assignment_loss‚0.234141495883988 /loss/FBt1/detection_loss‚3.4312641248106956 /loss/FBt2/detection_loss‚0.8050789222093847 1loss/SRqqt3/detection_loss‚1.8375253296560712 /loss/SRqqt4/detection_loss‚2.03121838505779 /loss/FRt1/assignment_loss‚4.185014825118215 2loss/SRqqt2/assignment_loss‚1.7083660011524207 .loss/FRt4/detection_loss‚2.529040356477102 /loss/FBt4/detection_loss‚3.7547284364700317 %loss/total_loss‚55.02374803726308 $ _timestamp‚1.788418111477546e+09 .loss/FRt2/assignment_loss‚2.56710892702852†ÎëX›Ü‚(: Ç°äÔìĹx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥×á«hÝ‚(:Ý Ç°äÔû–Â{ network.sent‚ 2.76988e+06) disk./.usagePercent‚73.69028772748216 disk.nvme0c0n1.in‚ 4.9296875 disk.sda3.in‚7.1875 disk.dm-1.out‚153.125 disk.rbd0.in‚0 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2841.0859375 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.23828125 disk.sda.out‚ 153.1640625 disk.rbd1.out‚ 0.67626953125 disk.nvme0n1.in‚ 4.9296875 disk.dm-0.in‚ 4.9296875 disk.dm-1.in‚7.1875 disk.rbd4.in‚0' disk.nvme0c0n1.out‚6566.39697265625 disk.rbd1.in‚0& proc.memory.availableMB‚ 5.87109375 disk.sda1.out‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚6566.39697265625 disk.drbd1777.in‚ 1.03515625$ memory_percent‚99.96416568756104 disk.sda3.out‚153.125 cpu‚25.225662738392025 proc.cpu.threads‚17 disk.sda1.in‚ 0.05078125 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 network.recv‚604413 disk.rbd0.out‚0* proc.memory.percent‚17.340612411499023% disk./.usageGB‚1098.0385398864746" disk.dm-0.out‚6566.39697265625 disk.rbd2.in‚0 disk.rbd2.out‚ 93.5283203125CJ°mÞ‚(:… Ö°äÔ¦”x gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-Â|Ÿsß‚(:è Ö°äÔºæ£{ disk.rbd0.out‚0" disk.dm-0.out‚6665.58837890625 disk.sda2.out‚ 0.0390625* proc.memory.percent‚17.340612411499023% disk./.usageGB‚1098.0385437011719 disk.drbd1777.in‚ 1.03515625% disk.nvme0n1.out‚6665.58837890625 network.sent‚ 2.877844e+06) disk./.usagePercent‚73.69028798348968 disk.dm-1.in‚7.1875 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.nvme0n1.in‚ 4.9296875# memory_percent‚99.9643087387085 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 0.67626953125" proc.memory.rssMB‚ 2841.0859375 disk.rbd4.out‚0 disk.sda.in‚ 7.23828125 disk.sda.out‚ 153.58203125 disk.dm-0.in‚ 4.9296875 disk.dm-1.out‚ 153.54296875 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 cpu‚25.224055059210794 disk.rbd2.out‚ 95.1650390625' disk.nvme0c0n1.out‚6665.58837890625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚7.1875 disk.sda3.out‚ 153.54296875& proc.memory.availableMB‚ 5.84765625 proc.cpu.threads‚17 network.recv‚6103244Ùà‚(:… å°äԘ؅x gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 n?ná‚(:ã å°äÔŒ»æz disk.rbd2.out‚ 95.1806640625 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚0$ memory_percent‚99.96275901794434) disk./.usagePercent‚73.69028977554242 disk.sda1.out‚0" disk.dm-0.out‚6666.16162109375 disk.dm-1.out‚ 153.9296875# proc.memory.rssMB‚ 2841.08984375% disk./.usageGB‚1098.0385704040527 disk.rbd3.out‚0 disk.sda3.out‚ 153.9296875 disk.rbd1.out‚ 0.67626953125% disk.nvme0n1.out‚6666.16162109375 disk.sda2.out‚ 0.0390625 cpu‚25.22248476182203 disk.rbd4.out‚0 disk.sda.out‚ 153.96875 disk.nvme0c0n1.in‚ 4.9296875 network.sent‚ 2.884139e+06 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda3.in‚7.1875 disk.dm-0.in‚ 4.9296875 disk.sda2.in‚0 disk.dm-1.in‚7.1875 disk.sda.in‚ 7.23828125 disk.nvme0n1.in‚ 4.9296875% proc.memory.availableMB‚ 6.1015625 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625' disk.nvme0c0n1.out‚6666.16162109375 disk.sda1.in‚ 0.05078125* proc.memory.percent‚17.340636253356934 network.recv‚615013Š72â- _wandbruntime‚1626  _runtime‚1626åèyºÄã¾ _step‚135 % lr-AdamW/pg1‚0.000798310772628422 % lr-AdamW/pg2‚0.000798310772628422 trainer/global_step‚1699 $ _timestamp‚1.788418155707882e+09 _runtime‚1626.457924702¡ §iÉäà % lr-AdamW/pg2‚0.000798310772628422 trainer/global_step‚1699 $ _timestamp‚1.788418155707882e+09 _runtime‚1626.457924702 _step‚135 % lr-AdamW/pg1‚0.000798310772628422‡qÅs-2å- _wandbruntime‚1626  _runtime‚1626÷@AµÇ'æÁO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66838439719936 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71730144849920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63286834637824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3139097447215 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11357629 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66838421716992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚69986648657920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29536483 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44061857 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75137278 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31075423 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3208666361856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11356981 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44087525 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74939086325760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66838439719936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3148455143424 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74938787271680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44061857 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66838421716992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71730120909824 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63286817598464 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18179504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69986648657920 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3148226940928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3139321096355 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1626.458789251 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3148455143424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18179502 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3208941475840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69986894863360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31075421 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69986894863360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 % _timestamp‚1.7884181557086375e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31075423 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299054080 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚66426155734179 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66425915045679 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3148226940928 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3208666361856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11356981 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44087525 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74939086325760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66838439719936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3148455143424 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74938787271680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44061857 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66838421716992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71730120909824 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63286817598464 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18179504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69986648657920 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3148226940928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3139321096355 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1626.458789251 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3148455143424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18179502 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3208941475840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69986894863360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31075421 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69986894863360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 % _timestamp‚1.7884181557086375e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31075423ˆ#:_A2è- _wandbruntime‚1627  _runtime‚1627ZŒá\«éËM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66464049903135 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31093142 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75206244 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44113102 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44087408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71771290606080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3150292851200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66464290591843 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71771314546176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3210828189184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63323137095680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75180548 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on‹Œæ&_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3141153496163 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75206244 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75180548 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29554429 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3140929846815 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚66876777664000 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3150064635392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63323120056320 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3210548894208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29553763 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44113102 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚66876777664000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18189857 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303235072 _step‚137 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70027070515200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74981839500288 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3150064635392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚1627.310527275 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31093140 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3150292851200 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70027070515200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66876759661056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11364570 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44087408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚66876759661056 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31093142 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11363906 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18189859 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 % _timestamp‚1.7884181565599964e+09¬zŽÖ&êÐM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71771314546176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3210828189184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63323137095680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75180548 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3141153496163 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75206244 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75180548 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29554429 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3140929846815 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚66876777664000 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3150064635392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63323120056320 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3210548894208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29553763 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44113102 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚66876777664000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18189857 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303235072 _step‚137 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70027070515200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74981839500288 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3150064635392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚1627.310527275 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31093140 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3150292851200 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70027070515200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66876759661056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11364570 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44087408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚66876759661056 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31093142 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11363906 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18189859 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 % _timestamp‚1.7884181565599964e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66464049903135 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31093142 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75206244 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44113102 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44087408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71771290606080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3150292851200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66464290591843 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0‰WMÆ 2ë- _wandbruntime‚1627  _runtime‚1627ÆÙYìÓ 1loss/FBt2/assignment_loss‚0.18704538580421493 0loss/FBt3/assignment_loss‚1.1789114475250244 /loss/FRt1/detection_loss‚2.6293083652853966 trainer/global_step‚1699 /loss/FRt4/assignment_loss‚4.578331227960257 .loss/FBt1/detection_loss‚2.253050312399864 0loss/SRqqt3/detection_loss‚1.906846010341094 0loss/SRqqt4/detection_loss‚2.175061748141334 /loss/FBt4/detection_loss‚2.9289445281028748 /loss/FRt2/detection_loss‚1.6834991447351597 /loss/FRt3/detection_loss‚2.4091370326021444 /loss/FBt2/detection_loss‚0.7084622779792106 % _timestamp‚1.7884181565617523e+09 1loss/SRqqt1/assignment_loss‚2.533903729915619 /loss/FBt1/assignment_loss‚0.509323617280946 &loss/total_loss‚54.183635825510244 _runtime‚1627.311590969 _step‚138 0loss/FRt1/assignment_loss‚4.3044485539197925 /loss/FRt3/assignment_loss‚4.057595760925956 2loss/SRqqt2/assignment_loss‚1.7813089863728668 0loss/FBt4/assignment_loss‚0.5753860902623273 1loss/SRqqt1/detection_loss‚2.7593868708610536 2loss/SRqqt3/assignment_loss‚2.0124090480116696 2loss/SRqqt4/assignment_loss‚1.9194776699656533 1loss/SRqqt2/detection_loss‚1.4004541120918121 epoch‚1 .loss/FRt4/detection_loss‚2.597859526502675 /loss/FBt3/detection_loss‚5.1450324058532715 0loss/FRt2/assignment_loss‚1.9484519726700253œj…è^íØ 1loss/SRqqt1/detection_loss‚2.7593868708610536 2loss/SRqqt3/assignment_loss‚2.0124090480116696 2loss/SRqqt4/assignment_loss‚1.9194776699656533 1loss/SRqqt2/detection_loss‚1.4004541120918121 epoch‚1 .loss/FRt4/detection_loss‚2.597859526502675 /loss/FBt3/detection_loss‚5.1450324058532715 0loss/FRt2/assignment_loss‚1.9484519726700253 1loss/FBt2/assignment_loss‚0.18704538580421493 0loss/FBt3/assignment_loss‚1.1789114475250244 /loss/FRt1/detection_loss‚2.6293083652853966 trainer/global_step‚1699 /loss/FRt4/assignment_loss‚4.578331227960257 .loss/FBt1/detection_loss‚2.253050312399864 0loss/SRqqt3/detection_loss‚1.906846010341094 0loss/SRqqt4/detection_loss‚2.175061748141334 /loss/FBt4/detection_loss‚2.9289445281028748 /loss/FRt2/detection_loss‚1.6834991447351597 /loss/FRt3/detection_loss‚2.4091370326021444 /loss/FBt2/detection_loss‚0.7084622779792106 % _timestamp‚1.7884181565617523e+09 1loss/SRqqt1/assignment_loss‚2.533903729915619 /loss/FBt1/assignment_loss‚0.509323617280946 &loss/total_loss‚54.183635825510244 _runtime‚1627.311590969 _step‚138 0loss/FRt1/assignment_loss‚4.3044485539197925 /loss/FRt3/assignment_loss‚4.057595760925956 2loss/SRqqt2/assignment_loss‚1.7813089863728668 0loss/FBt4/assignment_loss‚0.5753860902623273ŠŸ4Íî‚(:„ ô°äÔ£óÌy gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.623 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;ý vpï‚(:å ô°äÔöÃÕz disk.rbd3.in‚0 disk.dm-0.in‚ 4.9296875 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 network.sent‚ 2.889456e+06 disk.dm-1.out‚ 154.5546875 disk.rbd2.in‚0 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚6765.30615234375 disk.sda1.out‚0 disk.sda.out‚ 154.59375 disk.sda3.out‚ 154.5546875 disk.sda3.in‚7.1875 disk.rbd1.in‚0 disk.rbd1.out‚ 0.67626953125# proc.memory.rssMB‚ 2841.08984375) disk./.usagePercent‚73.69028977554242 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.23828125$ memory_percent‚99.95837211608887% disk./.usageGB‚1098.0385704040527 disk.sda2.in‚0 disk.rbd2.out‚95.27490234375 disk.nvme0c0n1.in‚ 4.9296875 disk.nvme0n1.in‚ 4.9296875% disk.nvme0n1.out‚6765.30615234375 disk.rbd4.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 network.recv‚618535 disk.rbd4.out‚0% proc.memory.availableMB‚ 6.8203125* proc.memory.percent‚17.340636253356934 cpu‚25.220931498824246" disk.dm-0.out‚6765.30615234375 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.1875 disk.rbd0.in‚0Ç<®ð‚(:… ƒ±äÔíÿšx gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Âkñ‚(:à ƒ±äÔðŒ™z% disk.nvme0n1.out‚6864.38330078125 disk.dm-1.out‚ 155.0390625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚6864.38330078125 disk.rbd1.in‚0 network.sent‚ 2.997781e+06 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 0.67626953125* proc.memory.percent‚17.340660095214844 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.1875 disk.rbd2.out‚95.28662109375 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.dm-0.in‚ 4.9296875 disk.sda2.in‚0 disk.sda3.out‚ 155.0390625 cpu‚25.219267932384632) disk./.usagePercent‚73.69029796778352 disk.sda.out‚ 155.078125 disk.sda3.in‚7.1875 network.recv‚624926" disk.dm-0.out‚6864.38330078125 disk.rbd4.out‚0 disk.sda.in‚ 7.23828125% disk./.usageGB‚1098.0386924743652 disk.nvme0n1.in‚ 4.9296875 disk.rbd0.out‚0$ memory_percent‚99.95841979980469" proc.memory.availableMB‚6.8125 proc.memory.rssMB‚ 2841.09375 proc.cpu.threads‚17<·,@›ò‚(: ’±äÔïÛÉx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.171' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.834200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð­umó‚(:â ’±äÔ·½¡{ disk.dm-0.in‚ 4.9296875" disk.dm-0.out‚6864.91845703125 cpu‚25.217733530017806 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda2.out‚ 0.0390625$ memory_percent‚99.95994567871094 proc.memory.rssMB‚ 2841.09375 disk.dm-1.in‚7.1875 disk.dm-1.out‚ 155.62890625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda3.out‚ 155.62890625 disk.rbd1.in‚0 disk.rbd1.out‚ 0.67626953125 disk.nvme0n1.in‚ 4.9296875% disk.nvme0n1.out‚6864.91845703125) disk./.usagePercent‚73.69030616002462 disk.rbd4.in‚0* proc.memory.percent‚17.340660095214844 disk.rbd0.out‚0' disk.nvme0c0n1.out‚6864.91845703125 disk.sda.in‚ 7.23828125 disk.sda.out‚ 155.66796875 disk.sda3.in‚7.1875" proc.memory.availableMB‚6.5625 network.recv‚629045 disk.rbd2.out‚95.34912109375 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 network.sent‚ 3.0036e+06% disk./.usageGB‚1098.0388145446777Ovàµ2ô- _wandbruntime‚1671  _runtime‚1671‚1ÁWÆõÀ trainer/global_step‚1749 % _timestamp‚1.7884182006437669e+09 _runtime‚ 1671.39366368 _step‚139 & lr-AdamW/pg1‚0.0007974528443581253 & lr-AdamW/pg2‚0.0007974528443581253¯Î„EËöÅ & lr-AdamW/pg1‚0.0007974528443581253 & lr-AdamW/pg2‚0.0007974528443581253 trainer/global_step‚1749 % _timestamp‚1.7884182006437669e+09 _runtime‚ 1671.39366368 _step‚139‹;yÿ2÷- _wandbruntime‚1671  _runtime‚1671ÈI²‰´&øÁO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68332912198187 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45342759 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18697252 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11683441 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11682777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 trainer/global_step‚1749 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31961371 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3230731018423 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31961373 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚71995691203584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73788629662720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65101957530624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _runtime‚1671.394585366 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45368427 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68755336923136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45368427 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266725376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚71995444998144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3240354280448 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290665472 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65101940491264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77329800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3240126077952 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚71995444998144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77090692836864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31961371 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3240126077952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45342759 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚140 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3302353839616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30380029 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68755336923136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68755318920192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68755318920192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77090983502336 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31961373 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚71995444998144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3240354280448 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290665472 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65101940491264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77329800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3240126077952 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚71995444998144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77090692836864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31961371 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3240126077952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45342759 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚140 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3302353839616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30380029 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68755336923136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68755318920192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68755318920192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77090983502336 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31961373 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31979090 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3242191988224 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45394004 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30396688 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚77133729779200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 % _timestamp‚1.7884182017101321e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72035866855424 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73829775418880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3232563418023 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77373096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3303954360320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3232787067371 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚77134026722816 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68370806367143 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68793674867200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3241963772416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18707607 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3304227363840 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31979092 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68793674867200 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72035866855424 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31979090 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45368310 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚68793656864256 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45394004 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11689734 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77347400 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45368310 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273003520 trainer/global_step‚1749 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3241963772416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 _step‚141öd&Ö&üÐM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3242191988224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77347400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68371047055851 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65138242949120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18707609 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚72035620636672 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77373096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31979092 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77347400 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45368310 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273003520 trainer/global_step‚1749 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3241963772416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 _step‚141 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30397343 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68793656864256 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚1672.460724266 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31979090 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3242191988224 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45394004 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30396688 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚77133729779200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 % _timestamp‚1.7884182017101321e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72035866855424 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73829775418880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3232563418023 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77373096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3303954360320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3232787067371 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚77134026722816 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68370806367143 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68793674867200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3241963772416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18707607 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3304227363840 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31979092 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68793674867200 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72035866855424 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31979090 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45368310 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚68793656864256 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45394004 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11689734 Fkƒ‚(:à °±äÔöŸìz% disk.nvme0n1.out‚7063.94775390625 proc.cpu.threads‚17 network.recv‚638976% disk./.usageGB‚1098.0390586853027 disk.nvme0n1.in‚ 4.9296875 disk.nvme0c0n1.in‚ 4.9296875' disk.nvme0c0n1.out‚7063.94775390625 proc.memory.rssMB‚ 2841.09375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 3.117613e+06( disk./.usagePercent‚73.6903225445068 disk.rbd1.out‚ 0.67626953125 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0" disk.dm-0.out‚7063.94775390625 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0& proc.memory.availableMB‚ 6.73828125 disk.rbd1.in‚0 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 156.65625 disk.rbd4.in‚0 disk.sda.out‚ 156.6953125$ memory_percent‚99.95887279510498* proc.memory.percent‚17.340660095214844 disk.sda.in‚ 7.23828125 disk.sda3.in‚7.1875 disk.sda3.out‚ 156.65625 cpu‚25.214823219151267 disk.dm-1.in‚7.1875 disk.rbd2.out‚97.00146484375¥ÈèÉ„‚(:… ¿±äÔãÉ­x gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿC|q…‚(:æ ¿±äÔ‡ðÏz% disk.nvme0n1.out‚7163.35595703125 disk.dm-1.out‚ 157.49609375 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.1875 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.23828125 disk.rbd0.in‚0$ memory_percent‚99.96044635772705* proc.memory.percent‚17.340660095214844 cpu‚25.21340614101189 disk.sda.out‚ 157.53515625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9296875 disk.dm-0.in‚ 4.9296875' disk.nvme0c0n1.out‚7163.35595703125 disk.rbd3.out‚0& proc.memory.availableMB‚ 6.48046875 network.sent‚ 3.123628e+06 disk.sda3.in‚7.1875" disk.dm-0.out‚7163.35595703125 disk.rbd4.out‚0% disk./.usageGB‚1098.0390586853027 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0( disk./.usagePercent‚73.6903225445068 disk.sda3.out‚ 157.49609375 proc.memory.rssMB‚ 2841.09375 network.recv‚643474 disk.rbd1.out‚ 0.67626953125 disk.rbd2.out‚97.01708984375‚Dàù2†- _wandbruntime‚1716  _runtime‚1716ö7OmLJÁ & lr-AdamW/pg2‚0.0007965123291145675 trainer/global_step‚1799 % _timestamp‚1.7884182460116909e+09 _runtime‚1716.761635975 _step‚143 & lr-AdamW/pg1‚0.0007965123291145675|<ÔÎÌˆÆ _runtime‚1716.761635975 _step‚143 & lr-AdamW/pg1‚0.0007965123291145675 & lr-AdamW/pg2‚0.0007965123291145675 trainer/global_step‚1799 % _timestamp‚1.7884182460116909e+09gèmB2‰- _wandbruntime‚1716  _runtime‚1716•ï Æ'ŠÀO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74004241338368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46649329 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31223819 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3395733004800 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3322364589631 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19215002 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32847323 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70239427973695 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70672216123392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚70672234126336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12009492 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 288568320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74004241338368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19215004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12008817 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46649329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75847114475520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3332253417472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32847321 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79242558912000 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74004487543808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3332025214976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚70672234126336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 trainer/global_step‚1799 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 _runtime‚1716.762579894 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46623661 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3395468376576 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79242847480320 $ _timestamp‚1.788418246012454e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚70672216123392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31224496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79470982 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3322588238771 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚66917080423424 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1œë)YË'‹ÅO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74004241338368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19215004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12008817 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46649329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75847114475520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3332253417472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32847321 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79242558912000 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74004487543808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3332025214976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚70672234126336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 trainer/global_step‚1799 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 _runtime‚1716.762579894 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46623661 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3395468376576 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79242847480320 $ _timestamp‚1.788418246012454e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚70672216123392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31224496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79470982 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3322588238771 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚66917080423424 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74004241338368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46649329 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31223819 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3395733004800 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3322364589631 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19215002 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32847323 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70239427973695 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70672216123392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚70672234126336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12009492 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 288568320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0+z¦2Œ- _wandbruntime‚1717  _runtime‚1717@±Ð…9ÊM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74044416976896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3334091125248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46649212 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46674906 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3324420638579 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75888284171776 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3324196989231 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32865042 trainer/global_step‚1799 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79514252 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74044663195648 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74044663195648 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79539948 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79285594084864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚66953382881280 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46649212 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70710572070400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32865040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31241572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788418246874882e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12016213 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3397333853184 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79539948 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3397608953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70710554067456 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31240944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32865042 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46674906 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19225357 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275100672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3333862909440 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299040768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70277803519859 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚66953365841920 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚70710554067456 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3334091125248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12015587 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚70710572070400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32865040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 _runtime‚1717.625459694 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299[Í4,Õ&ŽÏM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79514252 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74044663195648 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74044663195648 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79539948 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79285594084864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚66953382881280 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46649212 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70710572070400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32865040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31241572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788418246874882e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12016213 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3397333853184 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79539948 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3397608953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70710554067456 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31240944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32865042 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46674906 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19225357 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275100672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3333862909440 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299040768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70277803519859 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚66953365841920 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚70710554067456 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3334091125248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12015587 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚70710572070400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32865040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 _runtime‚1717.625459694 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74044416976896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3334091125248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46649212 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46674906 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3324420638579 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75888284171776 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3324196989231 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32865042 trainer/global_step‚1799‘_ ?Å2- _wandbruntime‚1717  _runtime‚1717ƒvîÝQË 1loss/SRqqt3/detection_loss‚2.0908474654567484 0loss/SRqqt4/detection_loss‚2.273993712001377 /loss/FBt1/detection_loss‚1.9544210024178028 /loss/FRt4/assignment_loss‚4.549818832765926 /loss/FRt2/detection_loss‚1.7694825362414122 .loss/FRt3/detection_loss‚2.674606927343317 .loss/SRqqt1/detection_loss‚2.1761374771595 /loss/FBt4/detection_loss‚3.2789904922246933 %loss/total_loss‚52.29988122409691 _runtime‚1717.626845985 1loss/SRqqt3/assignment_loss‚2.197223243664722 0loss/FBt3/assignment_loss‚0.6626723602414131 .loss/FRt1/detection_loss‚2.423654857609007 /loss/FRt1/assignment_loss‚3.910964952574836 0loss/FRt2/assignment_loss‚2.3648819093193327 1loss/SRqqt4/assignment_loss‚2.026670374472936 0loss/FBt1/assignment_loss‚0.6048520924495051 % _timestamp‚1.7884182468769205e+09 0loss/FBt4/assignment_loss‚0.5821135943677973 /loss/FBt2/detection_loss‚0.6980839274137428 1loss/SRqqt1/assignment_loss‚2.310567295726608 1loss/SRqqt2/detection_loss‚1.4659431400624188 trainer/global_step‚1799 1loss/SRqqt2/assignment_loss‚1.515153543741414 _step‚146 /loss/FRt3/assignment_loss‚4.667656267011488 1loss/FBt2/assignment_loss‚0.19283851794316687 .loss/FRt4/detection_loss‚2.408087070692669 .loss/FBt3/detection_loss‚3.500219631195068 epoch‚1Û¸ºV‘Ð .loss/FBt3/detection_loss‚3.500219631195068 epoch‚1 1loss/SRqqt3/detection_loss‚2.0908474654567484 0loss/SRqqt4/detection_loss‚2.273993712001377 /loss/FBt1/detection_loss‚1.9544210024178028 /loss/FRt4/assignment_loss‚4.549818832765926 /loss/FRt2/detection_loss‚1.7694825362414122 .loss/FRt3/detection_loss‚2.674606927343317 .loss/SRqqt1/detection_loss‚2.1761374771595 /loss/FBt4/detection_loss‚3.2789904922246933 %loss/total_loss‚52.29988122409691 _runtime‚1717.626845985 1loss/SRqqt3/assignment_loss‚2.197223243664722 0loss/FBt3/assignment_loss‚0.6626723602414131 .loss/FRt1/detection_loss‚2.423654857609007 /loss/FRt1/assignment_loss‚3.910964952574836 0loss/FRt2/assignment_loss‚2.3648819093193327 1loss/SRqqt4/assignment_loss‚2.026670374472936 0loss/FBt1/assignment_loss‚0.6048520924495051 % _timestamp‚1.7884182468769205e+09 0loss/FBt4/assignment_loss‚0.5821135943677973 /loss/FBt2/detection_loss‚0.6980839274137428 1loss/SRqqt1/assignment_loss‚2.310567295726608 1loss/SRqqt2/detection_loss‚1.4659431400624188 trainer/global_step‚1799 1loss/SRqqt2/assignment_loss‚1.515153543741414 _step‚146 /loss/FRt3/assignment_loss‚4.667656267011488 1loss/FBt2/assignment_loss‚0.19283851794316687 .loss/FRt4/detection_loss‚2.408087070692669’(潜’‚(:… αäÔ­¡Žx gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:,î$k“‚(:à αäÔù¶À{ cpu‚25.211980658424803 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.sda3.in‚7.1875& proc.memory.availableMB‚ 6.21484375( disk./.usagePercent‚73.6903225445068" disk.dm-0.out‚7163.78759765625 disk.dm-1.out‚ 158.1171875 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda.out‚ 158.15625 disk.nvme0n1.in‚ 4.9296875 proc.memory.rssMB‚ 2841.09375 network.sent‚ 3.129484e+06 disk.rbd2.out‚ 97.107421875 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.67626953125$ memory_percent‚99.96206760406494* proc.memory.percent‚17.340660095214844 disk.rbd2.in‚0 network.recv‚647369 disk.sda2.out‚ 0.0390625% disk./.usageGB‚1098.0390586853027 disk.drbd1777.in‚ 1.03515625' disk.nvme0c0n1.out‚7163.78759765625% disk.nvme0n1.out‚7163.78759765625 proc.cpu.threads‚17 disk.dm-0.in‚ 4.9296875 disk.sda2.in‚0 disk.sda.in‚ 7.23828125 disk.rbd0.out‚0 disk.sda3.out‚ 158.1171875 disk.dm-1.in‚7.1875 disk.rbd4.in‚0¼jÎÅ›”‚(: ݱäÔÄʼnx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ:°p•‚(:å ݱäÔ„“âz disk.sda1.in‚ 0.05078125 disk.sda2.out‚ 0.0390625$ memory_percent‚99.96063709259033 disk.rbd4.out‚0 disk.dm-0.in‚ 4.9296875 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚7263.09912109375 disk.rbd3.in‚0 disk.sda3.in‚7.1875 disk.rbd0.in‚0) disk./.usagePercent‚73.69032408055202 disk.rbd4.in‚0" disk.dm-0.out‚7263.09912109375 disk.dm-1.in‚7.1875 disk.dm-1.out‚ 158.56640625& proc.memory.availableMB‚ 6.44921875 cpu‚25.21062686098221% disk./.usageGB‚1098.0390815734863 disk.nvme0c0n1.in‚ 4.9296875 disk.sda3.out‚ 158.56640625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9296875% disk.nvme0n1.out‚7263.09912109375 disk.sda2.in‚0 network.sent‚ 3.237973e+06 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd2.in‚0 proc.memory.rssMB‚ 2841.09375 network.recv‚653895 disk.sda.out‚ 158.60546875* proc.memory.percent‚17.340660095214844 proc.cpu.threads‚17 disk.sda.in‚ 7.23828125 disk.rbd1.out‚ 0.76220703125 disk.rbd2.out‚ 97.123046875 ¿º–‚(:‚ ì±äÔ¬¸±x gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b}r¦`—‚(:Õ ì±äÔåîËz disk.rbd0.out‚0 network.sent‚ 3.242422e+06 disk.rbd3.out‚0 disk.rbd2.out‚ 97.248046875& proc.memory.availableMB‚ 6.43359375* proc.memory.percent‚17.340660095214844 cpu‚25.2091390818613 proc.cpu.threads‚17 disk.sda.in‚ 7.23828125 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0$ memory_percent‚99.96073246002197 proc.memory.rssMB‚ 2841.09375 disk.nvme0n1.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.1875 disk.rbd2.in‚0) disk./.usagePercent‚73.69044312405543 disk.rbd1.in‚0" disk.nvme0n1.out‚ 7364.08984375 disk.nvme0c0n1.in‚ 4.9296875$ disk.nvme0c0n1.out‚ 7364.08984375 disk.rbd3.in‚0 disk.dm-0.out‚ 7364.08984375 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 0.82470703125 disk.sda1.in‚ 0.05078125 disk.dm-0.in‚ 4.9296875 disk.dm-1.out‚ 159.1484375 network.recv‚656629% disk./.usageGB‚1098.0408554077148 disk.sda.out‚159.1875 disk.sda3.in‚7.1875 disk.sda3.out‚ 159.1484375 disk.rbd0.in‚0¦8 2˜- _wandbruntime‚1761  _runtime‚1761»?ó¤Å™¿ $ _timestamp‚1.788418291123032e+09 _runtime‚ 1761.87296327 _step‚147 & lr-AdamW/pg1‚0.0007954894228997154 & lr-AdamW/pg2‚0.0007954894228997154 trainer/global_step‚1849b[Í2ÊšÄ trainer/global_step‚1849 $ _timestamp‚1.788418291123032e+09 _runtime‚ 1761.87296327 _step‚147 & lr-AdamW/pg1‚0.0007954894228997154 & lr-AdamW/pg2‚0.0007954894228997154“ÒO•F2›- _wandbruntime‚1761  _runtime‚1761°G¤Ç'œÁO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72589113326592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47930231 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81394453761536 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3423924352000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 _runtime‚1761.873887671 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3423924352000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303248384 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72589131329536 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚81394757009920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32065012 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72146184437703 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68732203316224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76013037678592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3488878413312 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81637834 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47904563 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72146425126203 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72589113326592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19732754 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76013283884032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72589131329536 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚77905599288320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33733271 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47930231 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81663504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12331606 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 % _timestamp‚1.7884182911237495e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76013037678592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3424152554496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68732186276864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33733273 _step‚148 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47904563 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚654 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32064358 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3424152554496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33733271 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚77905575348224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76013283884032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚652 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279308288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-10¸*ð—ÆO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72589113326592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19732754 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76013283884032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72589131329536 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚77905599288320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33733271 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47930231 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81663504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12331606 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 % _timestamp‚1.7884182911237495e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76013037678592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3424152554496 HBDeviceStatsMonitor.on_traint®%ö5"_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68732186276864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33733273 _step‚148 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47904563 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚654 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32064358 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3424152554496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33733271 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚77905575348224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76013283884032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚652 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279308288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72589113326592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47930231 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81394453761536 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3423924352000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 _runtime‚1761.873887671 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3423924352000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303248384 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72589131329536 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚81394757009920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32065012 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72146184437703 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68732203316224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76013037678592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3488878413312 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81637834 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47904563 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72146425126203 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584”>Êž¯2ž- _wandbruntime‚1762  _runtime‚1762Ì¥¿ Ñ&ŸËM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3425990262272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19743107 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72184319295159 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3416054209787 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76053459535872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47955808 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32081590 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚1762.720976515 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72627451270656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33750990 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3425990262272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47930114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚81437487687168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3425762046464 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81706800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19743109 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81437790922240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68768488734720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47955808 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81706800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72627469273600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76053213317120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81681104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3425762046464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3490742642688 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3415830560439 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32080927 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76053459535872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3491021937664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303235072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72184559983867 _step‚149 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72627469273600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12337820 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1849 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294976 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624¨/£Ö& ÐM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76053459535872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3491021937664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303235072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72184559983867 _step‚149 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72627469273600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12337820 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1849 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294976 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3425990262272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19743107 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72184319295159 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3416054209787 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76053459535872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47955808 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32081590 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚1762.720976515 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72627451270656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33750990 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3425990262272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47930114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚81437487687168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3425762046464 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81706800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19743109 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81437790922240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68768488734720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47955808 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81706800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72627469273600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76053213317120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81681104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3425762046464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3490742642688 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3415830560439 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32080927 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824•¢f2¡- _wandbruntime‚1762  _runtime‚1762¢©ÔMZ¢Ô 2loss/SRqqt3/assignment_loss‚2.0725752260626815 .loss/FRt2/detection_loss‚2.121263102014014 /loss/FBt1/detection_loss‚2.1247949972748756 .loss/FBt4/detection_loss‚2.463969016075134 0loss/FBt4/assignment_loss‚0.6364090621471405 1loss/SRqqt4/detection_loss‚2.0410202882356114 /loss/FBt3/detection_loss‚3.6479192972183228 trainer/global_step‚1849 /loss/FRt4/assignment_loss‚4.567943304084068 1loss/SRqqt4/assignment_loss‚2.455904307188811 %loss/total_loss‚53.09438438084669 0loss/FRt3/assignment_loss‚3.8224776664445566 1loss/FBt2/assignment_loss‚0.17809514225738346 1loss/SRqqt1/detection_loss‚2.4999878630042076 epoch‚1 $ _timestamp‚1.788418291972264e+09 2loss/SRqqt2/assignment_loss‚1.7218712198237578 0loss/FBt1/assignment_loss‚0.8397138629640851 .loss/FRt3/detection_loss‚2.456004790095396 0loss/SRqqt3/detection_loss‚2.006545076283013 _runtime‚1762.722128446 0loss/FRt1/assignment_loss‚4.4558767766663525 1loss/FBt3/assignment_loss‚0.35684650111943483 /loss/FRt1/detection_loss‚2.8796047261266997 1loss/SRqqt2/detection_loss‚1.2149080127477645 0loss/FRt2/assignment_loss‚3.0227168519446193 2loss/SRqqt1/assignment_loss‚2.5222293585538864 /loss/FRt4/detection_loss‚2.3266667155332343 /loss/FBt2/detection_loss‚0.6590412169816378 _step‚150•Ã›_£Ù 0loss/FBt4/assignment_loss‚0.6364090621471405 1loss/SRqqt4/detection_loss‚2.0410202882356114 /loss/FBt3/detection_loss‚3.6479192972183228 trainer/global_step‚1849 /loss/FRt4/assignment_loss‚4.567943304084068 1loss/SRqqt4/assignment_loss‚2.455904307188811 %loss/total_loss‚53.09438438084669 0loss/FRt3/assignment_loss‚3.8224776664445566 1loss/FBt2/assignment_loss‚0.17809514225738346 1loss/SRqqt1/detection_loss‚2.4999878630042076 epoch‚1 $ _timestamp‚1.788418291972264e+09 2loss/SRqqt2/assignment_loss‚1.7218712198237578 0loss/FBt1/assignment_loss‚0.8397138629640851 .loss/FRt3/detection_loss‚2.456004790095396 0loss/SRqqt3/detection_loss‚2.006545076283013 _runtime‚1762.722128446 0loss/FRt1/assignment_loss‚4.4558767766663525 1loss/FBt3/assignment_loss‚0.35684650111943483 /loss/FRt1/detection_loss‚2.8796047261266997 1loss/SRqqt2/detection_loss‚1.2149080127477645 0loss/FRt2/assignment_loss‚3.0227168519446193 2loss/SRqqt1/assignment_loss‚2.5222293585538864 /loss/FRt4/detection_loss‚2.3266667155332343 /loss/FBt2/detection_loss‚0.6590412169816378 _step‚150 2loss/SRqqt3/assignment_loss‚2.0725752260626815 .loss/FRt2/detection_loss‚2.121263102014014 /loss/FBt1/detection_loss‚2.1247949972748756 .loss/FBt4/detection_loss‚2.463969016075134–bb s¤‚(:… û±äÔÔ“½x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.867' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9734 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ŽR…8¥‚(:Ú û±äÔ²“í{ disk.rbd2.in‚0! disk.dm-0.out‚7463.5869140625 disk.sda2.in‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.sda3.in‚7.1875 disk.dm-1.in‚7.1875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd1.out‚ 0.82470703125 disk.nvme0n1.in‚ 4.9296875# memory_percent‚99.9596357345581 disk.dm-0.in‚ 4.9296875 disk.rbd3.out‚0 disk.dm-1.out‚ 159.6171875 disk.rbd2.out‚ 98.775390625 network.sent‚ 3.248494e+06 network.recv‚661288% disk./.usageGB‚1098.0408172607422 disk.drbd1777.in‚ 1.03515625$ disk.nvme0n1.out‚7463.5869140625 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd3.in‚0 disk.sda.out‚ 159.65625 disk.rbd1.in‚0 proc.memory.rssMB‚ 2841.09375 disk.rbd0.in‚0 disk.rbd4.in‚0& diskµ—w-.nvme0c0n1.out‚7463.5869140625 disk.sda.in‚ 7.23828125 disk.sda2.out‚ 0.0390625& proc.memory.availableMB‚ 6.61328125* proc.memory.percent‚17.340660095214844 cpu‚25.2077737058635( disk./.usagePercent‚73.6904405639801 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 159.6171875(êr¦‚(:… Š²äÔŒõºx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÈ«%[§‚(:Ð Š²äÔ˜„·{ disk.sda1.in‚ 0.05078125# memory_percent‚99.9603271484375 cpu‚25.20634131705644 disk.sda.in‚ 7.23828125 disk.sda.out‚ 160.2890625 disk.rbd1.in‚0 disk.rbd2.out‚ 98.900390625 network.recv‚666978 disk.rbd1.out‚ 0.85693359375 disk.rbd0.in‚0' disk.nvme0c0n1.out‚7463.98095703125 disk.sda1.out‚0% disk./.usageGB‚1098.0408363342285 disk.sda3.out‚160.25 disk.drbd1777.in‚ 1.03515625 proc.memory.availableMB‚6.5 proc.cpu.threads‚17) disk./.usagePercent‚73.69044184401776 disk.sda3.in‚7.1875 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0 disk.rbd4.out‚0 proc.memory.rssMB‚ 2841.09375* proc.memory.percent‚17.340660095214844 disk.rbd3.in‚0% disk.nvme0n1.out‚7463.98095703125 disk.dm-1.out‚160.25 network.sent‚ 3.356442e+06 disk.nvme0n1.in‚ 4.9296875" disk.dm-0.out‚7463.98095703125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd3.out‚0 disk.dm-1.in‚7.1875Gx ©›¨‚(: ™²äÔ‘É”x gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.031' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.406200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OѸÛq©‚(:æ ™²äÔÖú°z disk.rbd0.out‚0 network.recv‚671814 disk.sda.out‚ 160.76953125) disk./.usagePercent‚73.69044184401776 disk.nvme0n1.in‚ 4.9296875 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9296875' disk.nvme0c0n1.out‚7563.40869140625 disk.sda1.out‚0$ memory_percent‚99.96039867401123 disk.rbd3.in‚0% disk./.usageGB‚1098.0408363342285 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.340660095214844 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 network.sent‚ 3.362724e+06 disk.dm-1.out‚ 160.73046875 disk.rbd2.out‚ 98.916015625 disk.drbd1777.in‚ 1.03515625 disk.sda.in‚ 7.23828125 cpu‚25.205087558339002 disk.sda3.out‚ 160.73046875 proc.cpu.threads‚17 disk.rbd3.out‚0% disk.nvme0n1.out‚7563.40869140625& proc.memory.availableMB‚ 6.48828125" disk.dm-0.out‚7563.40869140625 disk.sda2.in‚0 disk.dm-1.in‚7.1875 disk.rbd2.in‚0 disk.sda3.in‚7.1875 proc.memory.rssMB‚ 2841.09375 disk.rbd1.out‚ 0.85693359375 disk.dm-0.in‚ 4.9296875 Ìý³2ª- _wandbruntime‚1806  _runtime‚1806ØhÀ:Ç«Á & lr-AdamW/pg1‚0.0007943843388856913 & lr-AdamW/pg2‚0.0007943843388856913 trainer/global_step‚1899 % _timestamp‚1.7884183362369866e+09 _runtime‚1806.986966994 _step‚151y]'þÌ¬Æ & lr-AdamW/pg1‚0.0007943843388856913 & lr-AdamW/pg2‚0.0007943843388856913 trainer/global_step‚1899 % _timestamp‚1.7884183362369866e+09 _runtime‚1806.986966994 _step‚151—ñ —©2­- _wandbruntime‚1806  _runtime‚1806¦¶_¹Æ'®ÀO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34619221 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79964084101120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78022080224256 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78021834018816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83804686 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74506010529792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20250504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270919680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1806.987866539 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74506010529792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3582540005888 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74053181590211 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3515823489024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294859776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚83546624107008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3582269086208 trainer/global_step‚1899 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3516051691520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78021834018816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49211133 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83830356 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79964060161024 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74506028532736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32904492 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚83546329247232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚70547326209024 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34619223 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788418336237708e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70547309169664ñ¾:Ë'¯ÅO XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78022080224256 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78021834018816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83804686 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74506010529792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20250504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270919680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1806.987866539 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74506010529792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3582540005888 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74053181590211 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3515823489024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294859776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚83546624107008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3582269086208 trainer/global_step‚1899 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3516051691520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78021834018816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49211133 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83830356 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79964060161024 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74506028532736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32904492 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚83546329247232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚70547326209024 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34619223 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788418336237708e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70547309169664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34619221 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79964084101120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561˜Þ.]2°- _wandbruntime‚1807  _runtime‚1807mryÇ ±ËM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3517661183488 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74544366476800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34636942 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12659598 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1899 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3517889399296 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70583628666880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12660239 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78062255876096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74544348473856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34636942 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34636940 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3584127845376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83873652 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80005253797376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚1807.843310485 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49236710 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3507687780995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3517661183488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3584407140352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32920455 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83847956 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78062255876096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303235072 4¾®Ï¿ .DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70583611627520 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83847956 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83873652 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49211016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20260859 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32920455 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83847956 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78062255876096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303235072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70583611627520 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83847956 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83873652 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49211016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20260859 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12659598 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1899 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3517889399296 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70583628666880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12660239 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78062255876096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74544348473856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34636942 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34636940 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3584127845376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83873652 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80005253797376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚1807.843310485 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49236710 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3507687780995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3517661183488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3584407140352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80030876564480 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85698510275584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50466367 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3607950828544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33749618 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50492035 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85698209124352 % _timestamp‚1.7884183815060546e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75959697365719 _runtime‚1852.256249033 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76422925735936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72362449101824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85997208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50492035 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚668 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50466367 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20768254 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82022568913920 trainer/global_step‚1949 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72362432062464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76422907732992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12982032 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3675664150528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚80030876564480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35505171 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301151232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3597488952395 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35505173 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12981366 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20768252 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277211136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35505173 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚156 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚80030630359040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35505171 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚75959938054219 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80030630359040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ï\ßr" ÁÆO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚156 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚80030630359040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35505171 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚75959938054219 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80030630359040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76422925735936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3607950828544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3607722626048 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80030876564480 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85698510275584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50466367 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3607950828544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33749618 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50492035 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85698209124352 % _timestamp‚1.7884183815060546e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75959697365719 _runtime‚1852.256249033 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76422925735936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72362449101824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85997208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50492035 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚668 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50466367 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20768254 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82022568913920 trainer/global_step‚1949 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72362432062464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76422907732992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12982032 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3675664150528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚80030876564480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35505171 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301151232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3597488952395 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35505173 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12981366 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20768252 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277211136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35505173 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395œh3Ì\2Â- _wandbruntime‚1853  _runtime‚1853/Ö”Ñ&ÃËM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72398734520320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86014808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76461245677056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50517612 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50491918 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72398751559680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80070805997568 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82063738610176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3609788536320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35522892 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85741546744832 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 % _timestamp‚1.7884183823703425e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20778607 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3677537228288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3599097702855 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76461245677056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86040504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12987856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50491918 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86014808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3677808134656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12988490 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294846464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35522890 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3609560320512 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33766463 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20778609 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35522892 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1853.120664582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3609788536320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270906368 trainer/global_step‚1949 _step‚157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33767099 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86040504 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50517612 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75997832223175 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80071052216320 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76461263680000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3609560320512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065é~½eÖ&ÄÐM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86040504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12987856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50491918 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86014808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3677808134656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12988490 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294846464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35522890 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3609560320512 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33766463 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20778609 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35522892 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1853.120664582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3609788536320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270906368 trainer/global_step‚1949 _step‚157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33767099 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86040504 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50517612 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75997832223175 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80071052216320 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76461263680000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3609560320512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72398734520320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86014808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76461245677056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50517612 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50491918 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72398751559680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80070805997568 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82063738610176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3609788536320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35522892 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85741546744832 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 % _timestamp‚1.7884183823703425e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20778607 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3677537228288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3599097702855 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76461245677056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808“ò¦F2Å- _wandbruntime‚1853  _runtime‚1853.)>.VÆÐ 0loss/FRt1/assignment_loss‚3.8686048487822213 % _timestamp‚1.7884183823718722e+09 1loss/SRqqt1/assignment_loss‚2.484807163476944 0loss/FBt4/assignment_loss‚0.6523033668701019 .loss/FBt2/detection_loss‚0.717658702130842 .loss/FBt4/detection_loss‚2.464666611618466 epoch‚1 1loss/SRqqt3/assignment_loss‚2.043278082377381 .loss/FRt2/detection_loss‚2.000640564689449 0loss/FRt3/assignment_loss‚3.7382123985806026 2loss/SRqqt2/assignment_loss‚1.7510238603596957 2loss/SRqqt4/assignment_loss‚2.1439636701078557 /loss/FBt1/assignment_loss‚0.984837311009566 0loss/FRt4/assignment_loss‚3.5073590517044066 0loss/SRqqt1/detection_loss‚3.086556767423948 /loss/FBt1/detection_loss‚1.6271309504906337 trainer/global_step‚1949 _runtime‚1853.121719169 /loss/FRt2/assignment_loss‚2.709827832147187 0loss/FBt2/assignment_loss‚0.2572977490688841 /loss/FRt4/detection_loss‚2.4835963858498467 1loss/SRqqt3/detection_loss‚2.0161465958312705 0loss/SRqqt4/detection_loss‚1.683531709126572 /loss/FRt1/detection_loss‚2.4920769640141063 /loss/FRt3/detection_loss‚2.6821997890601286 1loss/SRqqt2/detection_loss‚1.2025011330842972 .loss/FBt3/detection_loss‚4.709836483001709 &loss/total_loss‚52.151537624137404 0loss/FBt3/assignment_loss‚0.8434796333312953 _step‚158˜à@ [ÇÕ /loss/FRt1/detection_loss‚2.4920769640141063 /loss/FRt3/detection_loss‚2.6821997890601286 1loss/SRqqt2/detection_loss‚1.2025011330842972 .loss/FBt3/detection_loss‚4.709836483001709 &loss/total_loss‚52.151537624137404 0loss/FBt3/assignment_loss‚0.8434796333312953 _step‚158 0loss/FRt1/assignment_loss‚3.8686048487822213 % _timestamp‚1.7884183823718722e+09 1loss/SRqqt1/assignment_loss‚2.484807163476944 0loss/FBt4/assignment_loss‚0.6523033668701019 .loss/FBt2/detection_loss‚0.717658702130842 .loss/FBt4/detection_loss‚2.464666611618466 epoch‚1 1loss/SRqqt3/assignment_loss‚2.043278082377381 .loss/FRt2/detection_loss‚2.000640564689449 0loss/FRt3/assignment_loss‚3.7382123985806026 2loss/SRqqt2/assignment_loss‚1.7510238603596957 2loss/SRqqt4/assignment_loss‚2.1439636701078557 /loss/FBt1/assignment_loss‚0.984837311009566 0loss/FRt4/assignment_loss‚3.5073590517044066 0loss/SRqqt1/detection_loss‚3.086556767423948 /loss/FBt1/detection_loss‚1.6271309504906337 trainer/global_step‚1949 _runtime‚1853.121719169 /loss/FRt2/assignment_loss‚2.709827832147187 0loss/FBt2/assignment_loss‚0.2572977490688841 /loss/FRt4/detection_loss‚2.4835963858498467 1loss/SRqqt3/detection_loss‚2.0161465958312705 0loss/SRqqt4/detection_loss‚1.683531709126572žª!çÈ‚(:… Õ²äÔÜ—³x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.163' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MíM$lÉ‚(:á Õ²äÔïàó{ network.sent‚ 3.488684e+06( disk./.usagePercent‚73.6904541323794 disk.rbd4.out‚0 disk.sda.in‚ 7.23828125* proc.memory.percent‚17.340660095214844 disk.nvme0n1.in‚ 4.9296875" disk.dm-0.out‚7861.42529296875 disk.sda2.out‚ 0.0390625! disk.rbd2.out‚100.59228515625 disk.dm-0.in‚ 4.9296875 disk.rbd3.out‚0 network.recv‚690581% disk./.usageGB‚1098.0410194396973 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 162.6484375$ memory_percent‚99.96225833892822' disk.nvme0c0n1.out‚7861.42529296875 disk.sda.out‚162.6875 disk.dm-1.out‚ 162.6484375 cpu‚25.19998021909532 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda3.in‚7.1875 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.in‚0% disk.nvme0n1.out‚7861.42529296875 disk.sda2.in‚0& proc.memory.availableMB‚ 6.18359375 proc.memory.rssMB‚ 2841.09375 disk.rbd1.in‚0 disk.rbd1.out‚ 0.98291015625 proc.cpu.threads‚17 disk.rbd4.in‚0 disk.dm-1.in‚7.1875 disk.rbd0.in‚0áãõÂÊ‚(:… ä²äÔÉó x gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.792' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±¹ŠÁË‚(:æ ä²äÔž¸Â{ disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 163.23046875 disk.sda3.out‚ 163.23046875 network.sent‚ 3.596193e+06' disk.nvme0c0n1.out‚7960.53076171875 disk.rbd3.out‚0 disk.sda3.in‚7.1875 disk.nvme0n1.in‚ 4.9296875 network.recv‚696113 disk.dm-1.in‚7.1875 disk.rbd1.in‚0% disk.nvme0n1.out‚7960.53076171875 cpu‚25.198807730390897( disk./.usagePercent‚73.6904541323794! disk.rbd2.out‚100.67041015625 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125# memory_percent‚99.9626636505127 proc.cpu.threads‚17% disk./.usageGB‚1098.0410194396973 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda.out‚ 163.26953125" disk.dm-0.out‚7960.53076171875 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.23828125 disk.dm-0.in‚ 4.9296875* proc.memory镹_°.percent‚17.340660095214844 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd1.out‚ 0.98291015625% proc.memory.availableMB‚ 6.1171875 proc.memory.rssMB‚ 2841.09375C§÷Ì‚(:… ó²äÔôàžx gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.872' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0± ¾utÍ‚(:é ó²äÔÈ£ò{ disk.sda2.in‚0 cpu‚25.197632849011757% disk.nvme0n1.out‚8059.67138671875 disk.rbd2.in‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 6.08203125) disk./.usagePercent‚73.69045822849995 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0 disk.rbd3.in‚0$ memory_percent‚99.96287822723389* proc.memory.percent‚17.340660095214844 disk.nvme0n1.in‚ 4.9296875! disk.rbd2.out‚100.67041015625 disk.rbd4.in‚0 proc.memory.rssMB‚ 2841.09375 network.recv‚700520 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.sda.out‚ 163.67578125 network.sent‚ 3.602136e+06" disk.dm-0.out‚8059.67138671875 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.23828125 disk.sda3.out‚ 163.63671875 disk.rbd1.in‚0 disk.dm-0.in‚ 4.9296875 disk.dm-1.in‚7.1875 disk.rbd4.out‚0' disk.nvme0c0n1.out‚8059.67138671875 disk.sda1.in‚ 0.05078125% disk./.usageGB‚1098.0410804748535 disk.rbd1.out‚ 0.98291015625 disk.dm-1.out‚ 163.63671875 disk.nvme0c0n1.in‚ 4.9296875 disk.sda3.in‚7.1875 proc.cpu.threads‚17x…æ2Î- _wandbruntime‚1897  _runtime‚1897ïqÈÇÏÁ & lr-AdamW/pg2‚0.0007919285757292717 trainer/global_step‚1999 % _timestamp‚1.7884184269339964e+09 _runtime‚1897.684043922 _step‚159 & lr-AdamW/pg1‚0.0007919285757292717ulÌÐÆ & lr-AdamW/pg2‚0.0007919285757292717 trainer/global_step‚1999 % _timestamp‚1.7884184269339964e+09 _runtime‚1897.684043922 _step‚159 & lr-AdamW/pg1‚0.0007919285757292717ŸàõÔ\2Ñ- _wandbruntime‚1897  _runtime‚1897ŸÌ—Ç'ÒÁO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34592282 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78339804936192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36391121 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚74177571994624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36391123 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82039672904704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21286004 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51747269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78339804936192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78339822939136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77866453829727 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87850442551296 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82039426699264 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13305623 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13306278 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51747269 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3769388824576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21286002 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3688898874463 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82039672904704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36391121 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3699621763072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77866694518227 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3699849965568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3699849965568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84081053726720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 % _timestamp‚1.7884184269347506e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _runtime‚1897.684932957 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303248384 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87850139302912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3689122523603 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84081029786624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279308288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88138390 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82039426699264 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51772937 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34591625 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 trainer/global_step‚1999 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3699621763072 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36391123 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82039672904704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36391121 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3699621763072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77866694518227 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3699849965568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3699849965568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84081053726720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 % _timestamp‚1.7884184269347506e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _runtime‚1897.684932957 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303248384 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87850139302912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3689122523603 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84081029786624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279308288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88138390 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82039426699264 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51772937 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34591625 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 trainer/global_step‚1999 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3699621763072 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36391123 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82039426699264 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13305623 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13306278 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51747269 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3769388824576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21286002 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3688898874463 Ì_o2Ô- _wandbruntime‚1898  _runtime‚1898O¯jÑ&ÕËM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3701459457536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21296357 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3701687673344 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13312131 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 _runtime‚1898.551121426 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294846464 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77904829375891 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36408840 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34608488 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87893178670080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88207356 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36408840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51798514 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3690731274063 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51772820 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88181660 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36408842 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82079848556544 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78378142880256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3771250093568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚1999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77904588687183 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74213874452480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74213857413120 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78378160883200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3701459457536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51772820 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0Ož9kÖÐM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KDDeviceStatsMon•ó¹3k&itor.on_train_batch_end/inactive_split.large_pool.peak‚23 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88207356 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36408842 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78378142880256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34608488 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87893178670080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88207356 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36408840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51798514 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3690731274063 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51772820 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88181660 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36408842 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82079848556544 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78378142880256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3771250093568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚1999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77904588687183 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74213874452480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74213857413120 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78378160883200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3701459457536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51772820 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3701459457536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21296357 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3701687673344 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13312131 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 _runtime‚1898.551121426 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294846464 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77904829375891 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36408840 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84048223039488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚79773210293735 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75992694887424 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53028171 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3791520900096 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚80256702139392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35432529 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279308288 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84048223039488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86139514599424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84048469244928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13628136 _step‚164 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75992677848064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80256720142336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639åz¤†åÅO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84048223039488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚79773210293735 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75992694887424 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53028171 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3791520900096 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚80256702139392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35432529 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279308288 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84048223039488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86139514599424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84048469244928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13628136 _step‚164 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75992677848064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80256720142336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3791749102592 GADeviceStatsMonitor.on_train_batch_stå+—‰Eart/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303248384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80256720142336 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37277071 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37277071 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 _runtime‚1943.199795682 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37277073 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13628775 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚79773450982235 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53053839 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3862886450176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3862607141888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166¤D´: 2æ- _wandbruntime‚1944  _runtime‚1944ÃóÑ&çËM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3793358594560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3793358594560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80295058086400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21814109 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚79811585839899 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35449345 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80295058086400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3782588494619 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3793586810368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3793586810368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37294792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13635236 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90045465670656 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76028997345280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90374208 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35448717 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37294790 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3864757434880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884184732983305e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37294792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53079416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76028980305920 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84088644896768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90045164532736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86180708235776 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _step‚165 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86180684295680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3782364845271 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53079416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13634610 trainer/global_step‚2049 _runtime‚1944.049094515 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80295040083456 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3864480237056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21814107 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90374208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84088644896768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53053722 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824„ ÈÖ&èÐM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80295058086400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3782588494619 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3793586810368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3793586810368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37294792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13635236 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90045465670656 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76028997345280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90374208 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35448717 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37294790 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3864757434880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884184732983305e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37294792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53079416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76028980305920 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84088644896768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90045164532736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86180708235776 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _step‚165 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86180684295680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3782364845271 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53079416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13634610 trainer/global_step‚2049 _runtime‚1944.049094515 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80295040083456 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3864480237056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21814107 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90374208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84088644896768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53053722 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3793358594560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3793358594560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80295058086400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21814109 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚79811585839899 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35449345 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0¥¡·²2é- _wandbruntime‚1944  _runtime‚1944‰= UêÏ /loss/FRt3/assignment_loss‚3.588322934779254 0loss/FBt3/assignment_loss‚0.2889578342437744 /loss/FBt4/detection_loss‚3.2062849193811416 0loss/FRt2/assignment_loss‚2.1219324323866102 1loss/SRqqt2/detection_loss‚0.9962553402435952 /loss/FRt2/detection_loss‚1.9739582354823748 0loss/SRqqt3/detection_loss‚2.072768004501567 1loss/SRqqt4/detection_loss‚1.9583377877871195 epoch‚1 &loss/total_loss‚49.431876235424056 % _timestamp‚1.7884184733004034e+09 0loss/FBt4/assignment_loss‚0.4915326450020075 0loss/SRqqt2/assignment_loss‚1.08845705560584 2loss/SRqqt3/assignment_loss‚1.8615190725700528 0loss/FBt2/assignment_loss‚0.3043287569477261 /loss/FBt2/detection_loss‚0.5254144514143303 _runtime‚1944.050242106 .loss/FRt4/detection_loss‚2.265202520749508 1loss/SRqqt1/detection_loss‚2.5921424258696404 _step‚166 /loss/FRt3/detection_loss‚2.5995366939089517 trainer/global_step‚2049 .loss/FBt1/detection_loss‚1.848912043230874 .loss/FBt3/detection_loss‚3.580979108810425 /loss/FRt1/detection_loss‚2.4220364078497276 /loss/FRt4/assignment_loss‚3.678470932520353 0loss/SRqqt1/assignment_loss‚2.23623998071018 2loss/SRqqt4/assignment_loss‚2.2355787330203585 0loss/FBt1/assignment_loss‚0.8359860679400819 /loss/FRt1/assignment_loss‚4.658721850468562¼ÕZëÔ /loss/FRt1/detection_loss‚2.4220364078497276 /loss/FRt4/assignment_loss‚3.678470932520353 0loss/SRqqt1/assignment_loss‚2.23623998071018 2loss/SRqqt4/assignment_loss‚2.2355787330203585 0loss/FBt1/assignment_loss‚0.8359860679400819 /loss/FRt1/assignment_loss‚4.658721850468562 /loss/FRt3/assignment_loss‚3.588322934779254 0loss/FBt3/assignment_loss‚0.2889578342437744 /loss/FBt4/detection_loss‚3.2062849193811416 0loss/FRt2/assignment_loss‚2.1219324323866102 1loss/SRqqt2/detection_loss‚0.9962553402435952 /loss/FRt2/detection_loss‚1.9739582354823748 0loss/SRqqt3/detection_loss‚2.072768004501567 1loss/SRqqt4/detection_loss‚1.9583377877871195 epoch‚1 &loss/total_loss‚49.431876235424056 % _timestamp‚1.7884184733004034e+09 0loss/FBt4/assignment_loss‚0.4915326450020075 0loss/SRqqt2/assignment_loss‚1.08845705560584 2loss/SRqqt3/assignment_loss‚1.8615190725700528 0loss/FBt2/assignment_loss‚0.3043287569477261 /loss/FBt2/detection_loss‚0.5254144514143303 _runtime‚1944.050242106 .loss/FRt4/detection_loss‚2.265202520749508 1loss/SRqqt1/detection_loss‚2.5921424258696404 _step‚166 /loss/FRt3/detection_loss‚2.5995366939089517 trainer/global_step‚2049 .loss/FBt1/detection_loss‚1.848912043230874 .loss/FBt3/detection_loss‚3.580979108810425¦:N£ì‚(:„ ¯³äÔ«ÕÀx gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.787' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eÉè÷kí‚(:à ¯³äÔÍäø{ disk.sda.in‚ 7.23828125 disk.sda.out‚ 165.671875 disk.sda3.out‚ 165.6328125 disk.rbd1.in‚0 disk.sda2.in‚0) disk./.usagePercent‚73.69046667674858 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.out‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚8357.72119140625 disk.dm-1.out‚ 165.6328125 disk.rbd2.out‚102.4208984375$ memory_percent‚99.96471405029297* proc.memory.percent‚17.340660095214844 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.rbd3.in‚0 cpu‚25.19302322079721 disk.rbd4.in‚0 disk.sda3.in‚7.1875 disk.sda2.out‚ 0.0390625 proc.memory.rssMB‚ 2841.09375 proc.cpu.threads‚17% disk./.usageGB‚1098.0412063598633 disk.dm-0.in‚ 4.9296875" disk.dm-0.out‚8357.72119140625# proc.memory.availableMB‚5.78125 network.sent‚ 3.727995e+06 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625' disk.nvme0c0n1.out‚8357.72119140625 disk.nvme0n1.in‚ 4.9296875 disk.dm-1.in‚7.1875 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 0.98291015625 disk.rbd2.in‚0 network.recv‚719879Òa$¼›î‚(: ¾³äÔÁ±x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.066' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.813200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â ³3hï‚(:Ý ¾³äÔ½âêz network.sent‚ 3.835954e+06 network.recv‚725611 disk.sda1.out‚0 disk.sda3.in‚7.1875 disk.nvme0n1.in‚ 4.9296875 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.98291015625 disk.dm-0.in‚ 4.9296875 disk.rbd4.in‚0 disk.sda3.out‚ 166.203125 proc.memory.availableMB‚6.25 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0% disk.nvme0n1.out‚8358.06884765625" disk.dm-0.out‚8358.06884765625 disk.sda2.out‚ 0.0390625$ memory_percent‚99.96185302734375 proc.memory.rssMB‚ 2841.09375* proc.memory.percent‚17.340660095214844 cpu‚25.19184938885154 disk.dm-1.in‚7.1875 disk.dm-1.out‚ 166.203125 disk.nvme0c0n1.in‚ 4.9296875 disk.sda2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.69043416379174% disk./.usageGB‚1098.0407218933105 disk.rbd0.in‚0 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚8358.06884765625! disk.rbd2.out‚102.58544921875 disk.sda.in‚ 7.23828125 disk.sda.out‚ 166.2421875¿uè=šð‚(: ͳäÔ¨‘‹x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.46' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.491999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ Œñ‚(:å ͳäÔ†ñz disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda.out‚ 167.16015625 proc.memory.rssMB‚ 2841.09375 network.recv‚730495 disk.rbd3.out‚0' disk./.usagePercent‚73.690436211852! disk.rbd2.out‚102.58544921875' disk.nvme0c0n1.out‚8457.16357421875 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.98291015625* proc.memory.percent‚17.340660095214844" disk.dm-0.out‚8457.16357421875 disk.dm-1.in‚7.1875 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.1875 disk.sda3.out‚ 167.12109375$ memory_percent‚99.96485710144043 network.sent‚ 3.842239e+06 disk.nvme0n1.in‚ 4.9296875 disk.sda2.in‚0 disk.rbd4.out‚0 disk.rbd3.in‚0% disk./.usageGB‚1098.0407524108887 disk.dm-0.in‚ 4.9296875 disk.dm-1.out‚ 167.12109375 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd1.in‚0% proc.memory.availableMB‚ 5.7578125 proc.cpu.threads‚17% disk.nvme0n1.out‚845B- –a7.16357421875 disk.sda2.out‚ 0.0390625 disk.sda.in‚ 7.23828125 cpu‚25.19080484046259·íH€2ò- _wandbruntime‚1988  _runtime‚19880Çâ˜ÇóÁ & lr-AdamW/pg1‚0.0007891470866480991 & lr-AdamW/pg2‚0.0007891470866480991 trainer/global_step‚2099 % _timestamp‚1.7884185178097599e+09 _runtime‚1988.559697536 _step‚167g„n5ÌôÆ & lr-AdamW/pg2‚0.0007891470866480991 trainer/global_step‚2099 % _timestamp‚1.7884185178097599e+09 _runtime‚1988.559697536 _step‚167 & lr-AdamW/pg1‚0.0007891470866480991§L,"š2õ- _wandbruntime‚1988  _runtime‚1988[v¤’Ç'öÁO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13955889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82173599342592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13955253 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚81680207446243 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3883648239616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3883648239616 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚81679966757743 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚2099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3883420037120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82173617345536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88197999412224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3956347433984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38163021 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54309073 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86057265585152 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86057019379712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54334741 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92154075926528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92472094 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88198023352320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚77807817780224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 _runtime‚1988.560691168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54334741 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86057019379712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36277393 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚77807800740864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86057265585152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38163023 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54309073 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3956076514304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3872389666019 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294859776 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38163023 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92154370786304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3872166016879 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82173599342592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3883420037120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22321504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82173617345536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92497764 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88197999412224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3956347433984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38163021 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54309073 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86057265585152 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86057019379712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54334741 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92154075926528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92472094 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88198023352320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚77807817780224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 _runtime‚1988.560691168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54334741 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86057019379712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36277393 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚77807800740864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86057265585152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38163023 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54309073 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3956076514304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3872389666019 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294859776 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38163023 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92154370786304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3872166016879 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82173599342592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3883420037120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22321504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82173617345536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92497764 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚81680207446243 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262¨Óö2ø- _wandbruntime‚1989  _runtime‚1989¸£D\Ñ&ùËM ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92515364 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303235072 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚81718101615199 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1989.431165735 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3885485947392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82211937286656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3885485947392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92541060 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3885257731584 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54334624 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92197107525632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3885257731584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82211937286656 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13961422 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54334624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22331859 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13962044 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92197410760704 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884185186804757e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54360318 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54360318 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22331857 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 trainer/global_step‚2099 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3874222065827 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚82211955289600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92541060 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77844120238080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86097441236992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38180740 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3957938417152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92515364 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3873998416479 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36293279 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88239169108480 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0­Þ ÌúÐM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54360318 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54360318 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22331857 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 trainer/global_step‚2099 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3874222065827 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚82211955289600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92541060 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77844120238080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86097441236992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38180740 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3957938417152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92515364 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3873998416479 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36293279 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88239169108480 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92515364 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303235072 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚81718101615199 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1989.431165735 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3885485947392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82211937286656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3885485947392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92541060 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3885257731584 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54334624 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92197107525632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3885257731584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82211937286656 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13961422 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54334624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22331859 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13962044 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92197410760704 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884185186804757e+09©j¤—¼2û- _wandbruntime‚1989  _runtime‚1989ÅE¬YVüÐ &loss/total_loss‚51.359508730663514 /loss/FBt1/detection_loss‚2.7166936174035072 .loss/FBt3/detection_loss‚3.253227114677429 /loss/FBt4/detection_loss‚2.4988398949305215 _step‚170 2loss/SRqqt3/assignment_loss‚2.4100140434630375 trainer/global_step‚2099 2loss/SRqqt1/assignment_loss‚2.2960351378686967 0loss/SRqqt2/assignment_loss‚1.50801323926155 1loss/FBt3/assignment_loss‚0.03952039033174515 0loss/FBt4/assignment_loss‚0.5810841831068198 0loss/SRqqt3/detection_loss‚2.047479142533972 2loss/SRqqt4/assignment_loss‚2.0303594526545754 0loss/FBt1/assignment_loss‚0.8314367346465588 0loss/FBt2/assignment_loss‚0.3332444171352447 /loss/FRt1/detection_loss‚2.4710546176703203 /loss/FRt2/detection_loss‚1.9126239759397918 .loss/FRt4/detection_loss‚2.270894602245214 0loss/SRqqt1/detection_loss‚2.649948496972361 0loss/SRqqt2/detection_loss‚1.083768632735165 epoch‚1 0loss/FRt2/assignment_loss‚2.3650433626668206 1loss/SRqqt4/detection_loss‚1.8061167443620747 _runtime‚1989.432298589 .loss/FRt1/assignment_loss‚4.56233365337054 0loss/FRt3/assignment_loss‚3.6833780584511935 % _timestamp‚1.7884185186824694e+09 /loss/FRt4/assignment_loss‚4.085590090070452 .loss/FRt3/detection_loss‚3.046316586158894 /loss/FBt2/detection_loss‚0.8764925420070319®‘_([ýÕ &loss/total_loss‚51.359508730663514 /loss/FBt1/detection_loss‚2.7166936174035072 .loss/FBt3/detection_loss‚3.253227114677429 /loss/FBt4/detection_loss‚2.4988398949305215 _step‚170 2loss/SRqqt3/assignment_loss‚2.4100140434630375 trainer/global_step‚2099 2loss/SRqqt1/assignment_loss‚2.2960351378686967 0loss/SRqqt2/assignment_loss‚1.50801323926155 1loss/FBt3/assignment_loss‚0.03952039033174515 0loss/FBt4/assignment_loss‚0.5810841831068198 0loss/SRqqt3/detection_loss‚2.047479142533972 2loss/SRqqt4/assignment_loss‚2.0303594526545754 0loss/FBt1/assignment_loss‚0.8314367346465588 0loss/FBt2/assignment_loss‚0.3332444171352447 /loss/FRt1/detection_loss‚2.4710546176703203 /loss/FRt2/detection_loss‚1.9126239759397918 .loss/FRt4/detection_loss‚2.270894602245214 0loss/SRqqt1/detection_loss‚2.649948496972361 0loss/SRqqt2/detection_loss‚1.083768632735165 epoch‚1 0loss/FRt2/assignment_loss‚2.3650433626668206 1loss/SRqqt4/detection_loss‚1.8061167443620747 _runtime‚1989.432298589 .loss/FRt1/assignment_loss‚4.56233365337054 0loss/FRt3/assignment_loss‚3.6833780584511935 % _timestamp‚1.7884185186824694e+09 /loss/FRt4/assignment_loss‚4.085590090070452 .loss/FRt3/detection_loss‚3.046316586158894 /loss/FBt2/detection_loss‚0.8764925420070319ªŠ?šþ‚(: ܳäԿɲx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚88.993' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.798599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔZçPrÿ‚(:ç ܳäÔ³Ýz disk.sda2.in‚0 disk.rbd4.in‚0' disk.nvme0c0n1.out‚8556.40576171875 disk.sda1.out‚0 disk.rbd0.in‚0 proc.memory.rssMB‚ 2841.09375 disk.rbd3.in‚0 disk.nvme0n1.in‚ 4.9296875 cpu‚25.189746958811256 disk.drbd1777.out‚0 network.recv‚734101 disk.sda.out‚ 167.59765625 disk.dm-1.out‚ 167.55859375 disk.rbd2.in‚0! disk.rbd2.out‚102.66357421875 proc.cpu.threads‚17 network.sent‚ 3.847558e+06) disk./.usagePercent‚73.69043646785954 disk.dm-1.in‚7.1875 disk.rbd4.out‚0$ disk./.usageGB‚1098.040756225586 disk.sda.in‚ 7.23828125 disk.sda3.out‚ 167.55859375" disk.dm-0.out‚8556.40576171875 disk.sda1.in‚ 0.05078125 disk.dm-0.in‚ 4.9296875 disk.nvme0c0n1.in‚ 4.9296875* proc.memory.percent‚17.340660095214844 disk.rbd3.out‚0 disk.rbd1.out‚ 0.98291015625% disk.nvme0n1.out‚8556.40576171875 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0# memory_percent‚99.9640703201294& proc.memory.availableMB‚ 5.88671875 disk.drbd1777.in‚ 1.03515625 disk.sda3.in‚7.1875 disk.rbd1.in‚0Ô™\m€‚(:… ë³äԱ”x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê ƒxs‚(:è ë³äÔŠæÙ{) disk./.usagePercent‚73.69043646785954 disk.rbd4.in‚0' disk.nvme0c0n1.out‚8557.13037109375 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 network.recv‚740552 disk.drbd1777.in‚ 1.03515625 disk.rbd1.out‚ 0.98291015625% disk.nvme0n1.out‚8557.13037109375 disk.nvme0c0n1.in‚ 4.9296875 disk.dm-1.out‚ 168.01953125 disk.rbd0.out‚0 proc.memory.rssMB‚ 2841.09375 disk.nvme0n1.in‚ 4.9296875 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.sda.out‚ 168.05859375 disk.sda3.out‚ 168.01953125$ memory_percent‚99.96416568756104$ disk./.usageGB‚1098.040756225586 disk.sda2.in‚0 disk.dm-1.in‚7.1875! disk.rbd2.out‚102.66357421875* proc.memory.percent‚17.340660095214844 cpu‚25.188692756192175 proc.cpu.threads‚17 network.sent‚ 3.955791e+06 disk.dm-0.in‚ 4.9296875 disk.sda1.in‚ 0.05078125" disk.dm-0.out‚8557.13037109375 disk.rbd0.in‚0& proc.memory.availableMB‚ 5.87109375 disk.rbd3.in‚0 disk.sda.in‚ 7.23828125 disk.sda3.in‚7.1875Zþ‚¦›‚‚(: ú³äÔë·¬x gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.611' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.322200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Iºdyiƒ‚(:Þ ú³äÔ{' disk.nvme0c0n1.out‚8656.26318359375 disk.rbd3.out‚0 disk.sda.in‚ 7.23828125 disk.sda3.out‚ 168.59375 disk.rbd1.out‚ 0.98291015625 cpu‚25.18756980712123 disk.rbd0.in‚0) disk./.usagePercent‚73.69043262774653 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 168.59375 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0$ disk./.usageGB‚1098.040699005127! disk.rbd2.out‚104.25341796875 disk.sda.out‚ 168.6328125 disk.rbd1.in‚0* proc.memory.percent‚17.340707778930664 disk.rbd2.in‚0 network.sent‚ 3.961826e+06 network.recv‚744759 disk.dm-0.in‚ 4.9296875 disk.nvme0n1.in‚ 4.9296875% disk.nvme0n1.out‚8656.26318359375# memory_percent‚99.9643325805664 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9296875# proc.memory.availableMB‚5.84375 proc.cpu.threads‚17 disk.sda3.in‚7.1875" disk.dm-0.out‚8656.26318359375 disk.dm-1.in‚7.1875 disk.rbd4.out‚0" proc.memory.rssMB‚ 2841.1015625kÂâ^2„- _wandbruntime‚2033  _runtime‚2033“ãVjÇ…Á trainer/global_step‚2149 % _timestamp‚1.7884185629264932e+09 _runtime‚2033.676448134 _step‚171 & lr-AdamW/pg1‚0.0007876349088661575 & lr-AdamW/pg2‚0.0007876349088661575`Çë·Ì†Æ & lr-AdamW/pg1‚0.0007876349088661575 & lr-AdamW/pg2‚0.0007876349088661575 trainer/global_step‚2149 % _timestamp‚1.7884185629264932e+09 _runtime‚2033.676448134 _step‚171«µ{2‡- _wandbruntime‚2033  _runtime‚2033dazÇ'ˆÁO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84090514548736 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39048973 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84090496545792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79622940673024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55589975 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884185629272377e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88065815719936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3975319174144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83586963910251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37116865 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88065815719936 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83586723221751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚88066061925376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281405440 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3975319174144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79622923633664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3975547376640 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55589975 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39048971 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55615643 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3975547376640 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90256508165120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39048973 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94306297249792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94664616 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚94305991904256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90256484225024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3964023237227 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14278276 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3963799588087 trainer/global_step‚2149 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚663 _runtime‚2033.677320007 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39048971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4049507679232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚665 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94638946 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88066061925376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305345536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256>¨ŠTÒ‰ÆO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83586963910251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37116865 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88065815719936 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83586723221751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚88066061925376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281405440 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649140 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3975319174144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79622923633664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3975547376640 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55589975 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39048971 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55615643 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3975547376640 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90256508165120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39048973 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94306297249792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94664616 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚94305991904256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90256484225024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3964023237227 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14278276 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3963799588087 trainer/global_step‚2149 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚663 _runtime‚2033.677320007 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39048971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4049507679232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚665 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94638946 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88066061925376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305345536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84090514548736 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39048973 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84090496545792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79622940673024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55589975 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884185629272377e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88065815719936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3975319174144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395¬­r2Š- _wandbruntime‚2034  _runtime‚2034PwõÑ&‹ËM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84128852492800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4051648804352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84128834489856 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83624858079207 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94707912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14283793 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88106237577216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22849607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3965855637035 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55615526 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55641220 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83625098767915 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94682216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84128834489856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281392128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94682216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39066690 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚94349326665728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3977156868608 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39066690 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3977385084416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90297653921280 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79659226091520 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90297677861376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37134028 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 305332224 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3965631987687 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88106237577216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39066692 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22849609 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79659243130880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88105991358464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94707912 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55615526 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84128852492800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14284419 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚94349021333504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37133400 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3977385084416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3977156868608 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚173 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39066692 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2149$;çÉÖ&ŒÐM R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88105991358464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94707912 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55615526 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84128852492800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14284419 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚94349021333504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37133400 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3977385084416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3977156868608 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚173 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39066692 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2149 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84128852492800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4051648804352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84128834489856 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83624858079207 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94707912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14283793 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88106237577216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22849607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3965855637035 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55615526 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55641220 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83625098767915 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94682216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84128834489856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281392128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94682216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39066690 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚94349326665728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3977156868608 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39066690 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3977385084416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90297653921280 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79659226091520 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90297677861376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37134028 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 305332224 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3965631987687 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88106237577216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39066692 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22849609 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79659243130880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624­V³}2- _wandbruntime‚2034  _runtime‚20340¦ÊZŽÔ /loss/FRt4/assignment_loss‚3.845408943566409 /loss/FBt2/detection_loss‚0.7445263710687868 1loss/SRqqt3/detection_loss‚2.1685898741086325 /loss/FBt4/detection_loss‚3.1010587096214293 epoch‚1 trainer/global_step‚2149 /loss/FRt1/assignment_loss‚4.393198549747467 0loss/FBt1/assignment_loss‚0.6547885509207845 /loss/FRt4/detection_loss‚2.9779193473584726 _runtime‚2034.531847933 _step‚174 1loss/SRqqt4/assignment_loss‚1.849155105650425 0loss/FBt2/assignment_loss‚0.2747745925839019 1loss/SRqqt2/detection_loss‚1.0718714167467422 /loss/FBt3/detection_loss‚3.2026014626026154 0loss/FRt3/assignment_loss‚3.5893475568812825 2loss/SRqqt3/assignment_loss‚1.9358730183707342 /loss/FRt1/detection_loss‚2.4152652898803355 0loss/SRqqt4/detection_loss‚1.901452324554032 %loss/total_loss‚51.08351707165599 % _timestamp‚1.7884185637820263e+09 1loss/SRqqt1/assignment_loss‚2.582756548314481 0loss/FBt4/assignment_loss‚0.5504972821101546 /loss/FRt2/detection_loss‚1.7990024500355428 0loss/FRt2/assignment_loss‚2.7253996231118025 1loss/FBt3/assignment_loss‚0.41912642028182745 .loss/FRt3/detection_loss‚2.417174283577048 /loss/FBt1/detection_loss‚2.6288129199634898 2loss/SRqqt2/assignment_loss‚1.5927955395956006 1loss/SRqqt1/detection_loss‚2.2421208910039954*Ì&_Ù %loss/total_loss‚51.08351707165599 % _timestamp‚1.7884185637820263e+09 1loss/SRqqt1/assignment_loss‚2.582756548314481 0loss/FBt4/assignment_loss‚0.5504972821101546 /loss/FRt2/detection_loss‚1.7990024500355428 0loss/FRt2/assignment_loss‚2.7253996231118025 1loss/FBt3/assignment_loss‚0.41912642028182745 .loss/FRt3/detection_loss‚2.417174283577048 /loss/FBt1/detection_loss‚2.6288129199634898 2loss/SRqqt2/assignment_loss‚1.5927955395956006 1loss/SRqqt1/detection_loss‚2.2421208910039954 /loss/FRt4/assignment_loss‚3.845408943566409 /loss/FBt2/detection_loss‚0.7445263710687868 1loss/SRqqt3/detection_loss‚2.1685898741086325 /loss/FBt4/detection_loss‚3.1010587096214293 epoch‚1 trainer/global_step‚2149 /loss/FRt1/assignment_loss‚4.393198549747467 0loss/FBt1/assignment_loss‚0.6547885509207845 /loss/FRt4/detection_loss‚2.9779193473584726 _runtime‚2034.531847933 _step‚174 1loss/SRqqt4/assignment_loss‚1.849155105650425 0loss/FBt2/assignment_loss‚0.2747745925839019 1loss/SRqqt2/detection_loss‚1.0718714167467422 /loss/FBt3/detection_loss‚3.2026014626026154 0loss/FRt3/assignment_loss‚3.5893475568812825 2loss/SRqqt3/assignment_loss‚1.9358730183707342 /loss/FRt1/detection_loss‚2.4152652898803355 0loss/SRqqt4/detection_loss‚1.901452324554032®i†6¶™‚(:Ž ‰´äÔ„˜Çy gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.72' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.343999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼Óvq‘‚(:æ ‰´äÔèÔÈz disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚8755.41748046875 disk.sda1.out‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚8755.41748046875 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9296875$ memory_percent‚99.96449947357178 disk.dm-0.in‚ 4.9296875 disk.dm-1.in‚7.1875" proc.memory.rssMB‚ 2841.1015625 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.1875 network.sent‚ 3.968157e+06 disk.nvme0n1.in‚ 4.9296875 disk.rbd2.in‚0! disk.rbd2.out‚104.25341796875 disk.rbd0.out‚0 disk.sda.in‚ 7.23828125 cpu‚25.18658689026997 proc.cpu.threads‚17" disk.dm-0.out‚8755.41748046875 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda3.out‚ 169.1015625& proc.memory.availableMB‚ 5.81640625 network.recv‚749616) disk./.usagePercent‚73.69043416379174% disk./.usageGB‚1098.0407218933105 disk.rbd1.in‚0 disk.dm-1.out‚ 169.1015625 disk.sda.out‚ 169.140625* proc.memory.percent‚17.340707778930664 disk.rbd1.out‚ 0.98291015625 disk.rbd3.out‚0²I¸”’‚(:… ˜´äÔæ…²x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P8aj“‚(:ß ˜´äÔÜ‚˜| disk.sda2.out‚ 0.0390625 disk.dm-1.out‚169.6875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 169.7265625" proc.memory.rssMB‚ 2841.1015625 disk.nvme0n1.in‚ 4.9296875 disk.drbd1777.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17" disk.dm-0.out‚8854.69091796875 disk.rbd0.in‚0 cpu‚25.185441783477884 disk.rbd1.in‚0 disk.rbd1.out‚ 0.98291015625 disk.dm-0.in‚ 4.9296875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda3.out‚169.6875% disk./.usageGB‚1096.9442100524902 disk.rbd2.in‚0 disk.rbd2.out‚ 104.43359375 disk.nvme0c0n1.in‚ 4.9296875' disk.nvme0c0n1.out‚8854.69091796875$ memory_percent‚99.96302127838135% disk.nvme0n1.out‚8854.69091796875 disk.rbd4.in‚0 disk.sda.in‚ 7.23828125& proc.memory.availableMB‚ 6.05859375 network.recv‚755087) disk./.usagePercent‚73.61684633411957 disk.dm-1.in‚7.1875 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda3.in‚7.1875* proc.memory.percent‚17.340707778930664 network.sent‚ 4.075649e+06 disk.sda1.in‚ 0.05078125J?˜á”‚(:… §´äÔÆŸ™x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚107.171' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôf: i•‚(:Þ §´äÔï³¼{ disk.rbd4.in‚0 disk.sda.out‚170.125& proc.memory.availableMB‚ 6.05078125" proc.memory.rssMB‚ 2841.1015625 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 cpu‚25.184371575479844 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 170.0859375 disk.rbd0.out‚0 network.recv‚759540) disk./.usagePercent‚73.61680306884628! disk.dm-0.out‚9021.5068359375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda3.out‚ 170.0859375 disk.rbd1.out‚ 0.98291015625 disk.rbd4.out‚0$ memory_percent‚99.96306896209717 disk.dm-1.in‚7.1875& disk.nvme0c0n1.out‚9021.5068359375 disk.sda.in‚ 7.23828125 disk.sda3.in‚7.1875* proc.memory.percent‚17.340707778930664 disk.nvme0n1.in‚ 4.9296875 disk.rbd2.out‚ 104.43359375 disk.rbd3.in‚0 network.sent‚ 4.081612e+06% disk./.usageGB‚1096.9435653686523$ disk.nvme0n1.out‚9021.5068359375 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.in‚ 0.05078125 disk.rbd2.in‚0>Å©Ð2–- _wandbruntime‚2078  _runtime‚2078*ÀMbÆ—À & lr-AdamW/pg2‚0.0007860421901539958 trainer/global_step‚2199 $ _timestamp‚1.788418607896922e+09 _runtime‚2078.646955617 _step‚175 & lr-AdamW/pg1‚0.0007860421901539958Ã'XV˘Š& lr-AdamW/pg1‚0.0007860421901539958 & lr-AdamW/pg2‚0.0007860421901539958 trainer/global_step‚2199 $ _timestamp‚1.788418607896922e+09 _runtime‚2078.646955617 _step‚175¯Ûi$k2™- _wandbruntime‚2078  _runtime‚2078ÍÌ mÒšÁO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚650 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚85493479685759 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4055433159295 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39934923 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4143266061312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4143007724544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56896545 RDDeviceSt˜IJ õ#atsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90074858265600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14602349 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39934923 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96458259039232 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96457976762368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90074612060160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4067218311168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92314969037824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 trainer/global_step‚2199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39934921 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23357002 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 258336768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81438063565824 % _timestamp‚1.7884186078976696e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚652 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37958701 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37958701 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90074858265600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14602349 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39934923 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96458259039232 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96457976762368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90074612060160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4067218311168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92314969037824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 trainer/global_step‚2199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39934921 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23357002 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 258336768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81438063565824 % _timestamp‚1.7884186078976696e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚652 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4069056005632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81474348984320 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96849068 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884186087696462e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96874764 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39952642 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39952640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299040768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4057489208243 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4069284221440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96501015965696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86045749696000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 _runtime‚2079.520267599 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92356138734080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4057265558895 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39952640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86045731693056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14609247 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _step‚177 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4069284221440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4145152332288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56896428 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86045749696000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39952642 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14608651 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚598 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚90115033917440 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56896428 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96849068 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56922122 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37976008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37976606 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23367357 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92356162674176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 trainer/global_step‚2199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86045731693056 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96874764 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81474366023680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23367359 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96501315006464 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚90115033917440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275100672 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96874764 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81474366023680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23367359 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96501315006464 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚90115033917440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275100672 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4069056005632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81474348984320 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96849068 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884186087696462e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96874764 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39952642 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@Ný½DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39952640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299040768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4057489208243 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4069284221440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96501015965696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86045749696000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 _runtime‚2079.520267599 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92356138734080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4057265558895 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39952640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86045731693056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14609247 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _step‚177 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4069284221440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4145152332288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56896428 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86045749696000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39952642 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14608651 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚598 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚90115033917440 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56896428 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96849068 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56922122 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37976008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37976606 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23367357 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92356162674176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299±ãŒvì2Ÿ- _wandbruntime‚2079  _runtime‚2079ì4x_Q Ë trainer/global_step‚2199 /loss/FBt4/detection_loss‚2.4290122787157693 epoch‚1 /loss/FRt3/assignment_loss‚3.574521142244339 0loss/FBt1/assignment_loss‚0.8034365892410278 /loss/FBt1/detection_loss‚2.3031039983034134 1loss/SRqqt1/detection_loss‚2.5745283870986015 _runtime‚2079.521331383 _step‚178 %loss/total_loss‚53.23445744933933 /loss/FRt1/assignment_loss‚4.248700763240005 .loss/FRt1/detection_loss‚2.678702767825488 /loss/FRt2/detection_loss‚2.3306678525275655 % _timestamp‚1.7884186087715049e+09 /loss/FRt4/assignment_loss‚4.214370639700639 2loss/SRqqt1/assignment_loss‚2.2623499379013525 1loss/SRqqt2/assignment_loss‚1.784893058737119 0loss/FBt4/assignment_loss‚0.9254846937126584 0loss/SRqqt3/detection_loss‚1.875644642348383 0loss/SRqqt4/assignment_loss‚2.14535121593559 0loss/FBt2/assignment_loss‚0.3050486686879003 0loss/FBt3/assignment_loss‚0.8761608004570007 .loss/FRt4/detection_loss‚2.422781368619517 1loss/SRqqt4/detection_loss‚1.9357708148788988 .loss/FBt3/detection_loss‚4.852244853973389 /loss/FRt2/assignment_loss‚2.414767801761627 1loss/SRqqt3/assignment_loss‚2.004031760727658 .loss/FRt3/detection_loss‚2.491259628534317 /loss/FBt2/detection_loss‚0.7401737738276521 0loss/SRqqt2/detection_loss‚1.041450010339419•¹ V¡Ð 0loss/FBt4/assignment_loss‚0.9254846937126584 0loss/SRqqt3/detection_loss‚1.875644642348383 0loss/SRqqt4/assignment_loss‚2.14535121593559 0loss/FBt2/assignment_loss‚0.3050486686879003 0loss/FBt3/assignment_loss‚0.8761608004570007 .loss/FRt4/detection_loss‚2.422781368619517 1loss/SRqqt4/detection_loss‚1.9357708148788988 .loss/FBt3/detection_loss‚4.852244853973389 /loss/FRt2/assignment_loss‚2.414767801761627 1loss/SRqqt3/assignment_loss‚2.004031760727658 .loss/FRt3/detection_loss‚2.491259628534317 /loss/FBt2/detection_loss‚0.7401737738276521 0loss/SRqqt2/detection_loss‚1.041450010339419 trainer/global_step‚2199 /loss/FBt4/detection_loss‚2.4290122787157693 epoch‚1 /loss/FRt3/assignment_loss‚3.574521142244339 0loss/FBt1/assignment_loss‚0.8034365892410278 /loss/FBt1/detection_loss‚2.3031039983034134 1loss/SRqqt1/detection_loss‚2.5745283870986015 _runtime‚2079.521331383 _step‚178 %loss/total_loss‚53.23445744933933 /loss/FRt1/assignment_loss‚4.248700763240005 .loss/FRt1/detection_loss‚2.678702767825488 /loss/FRt2/detection_loss‚2.3306678525275655 % _timestamp‚1.7884186087715049e+09 /loss/FRt4/assignment_loss‚4.214370639700639 2loss/SRqqt1/assignment_loss‚2.2623499379013525 1loss/SRqqt2/assignment_loss‚1.784893058737119²øC”›¢‚(: ¶´äÔª‡˜x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&‘}ßm£‚(:â ¶´äÔã¶Ýz disk.drbd1777.out‚0 network.sent‚ 4.087489e+06 disk.sda1.out‚0 disk.rbd2.out‚ 104.51171875 disk.rbd4.in‚0$ memory_percent‚99.96483325958252& proc.memory.availableMB‚ 5.76171875 proc.cpu.threads‚17$ disk./.usageGB‚1096.943733215332" disk.dm-0.out‚9021.98876953125 disk.dm-1.out‚ 170.6640625 disk.rbd2.in‚0 cpu‚25.183553569824696 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.sda3.in‚7.1875 disk.sda2.in‚0 disk.rbd4.out‚0" proc.memory.rssMB‚ 2841.1015625 disk.rbd3.out‚0 disk.sda.in‚ 7.23828125 disk.rbd1.out‚ 0.98291015625 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625* proc.memory.percent‚17.340707778930664( disk./.usagePercent‚73.6168143331778 disk.sda3.out‚ 170.6640625 disk.nvme0n1.in‚ 4.9296875 disk.dm-0.in‚ 4.9296875 disk.rbd0.out‚0 network.recv‚763392 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.1875 disk.nvme0c0n1.in‚ 4.9296875 disk.sda.out‚ 170.703125% disk.nvme0n1.out‚9021.98876953125' disk.nvme0c0n1.out‚9021.988769531254³žæŽ¤‚(:ƒ Å´äÔ¬Ò”x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËÎ^ej¥‚(:ß Å´äÔ¸ÃÄ{ disk.rbd4.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0) disk./.usagePercent‚73.61681561321546 disk.sda.in‚ 7.23828125 disk.sda3.in‚7.1875 disk.rbd2.out‚ 106.03125" proc.memory.rssMB‚ 2841.1015625 cpu‚25.182474263705874% disk./.usageGB‚1096.9437522888184 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.1875 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125* proc.memory.percent‚17.340707778930664 disk.rbd1.out‚ 0.98291015625 disk.sda2.in‚0 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0" disk.dm-0.out‚9022.58935546875% disk.nvme0n1.out‚9022.58935546875 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625 network.sent‚ 4.195948e+06 network.recv‚770043 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 4.9296875' disk.nvme0c0n1.out‚9022.58935546875$ memory_percent‚99.96490478515625 proc.memory.availableMB‚5.75 disk.sda.out‚ 171.20703125 disk.sda3.out‚ 171.16796875 disk.nvme0n1.in‚ 4.9296875 disk.dm-1.out‚ 171.16796875 disk.rbd0.in‚0'{ 0¦‚(:… Ô´äÔãÙšx gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r‚™£l§‚(:á Ô´äÔñåÐz disk.nvme0n1.in‚ 4.9296875 disk.sda1.out‚0% disk.nvme0n1.out‚9026.41455078125 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 106.10546875 disk.sda.out‚ 171.7109375 disk.rbd0.out‚0$ proc.memory.availableMB‚5.984375 cpu‚25.181568293034722 disk.sda3.out‚ 171.671875 disk.rbd0.in‚0* proc.memory.percent‚17.340707778930664 proc.cpu.threads‚17 network.sent‚ 4.200309e+06 network.recv‚772772 disk.rbd3.in‚0 disk.dm-1.in‚7.1875 disk.drbd1777.out‚0) disk./.usagePercent‚73.61702989152163 disk.rbd4.in‚0 disk.sda.in‚ 7.23828125 disk.rbd1.out‚ 0.98291015625% disk./.usageGB‚1096.9469451904297 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda3.in‚7.1875 disk.dm-0.in‚ 4.9296875 disk.rbd2.in‚0$ memory_percent‚99.96347427368164" proc.memory.rssMB‚ 2841.1015625' disk.nvme0c0n1.out‚9026.41455078125 disk.rbd1.in‚0" disk.dm-0.out‚9026.41455078125 disk.dm-1.out‚ 171.671875 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0iÓÕ2¨- _wandbruntime‚2123  _runtime‚2123Ý4Ç©Á % _timestamp‚1.7884186529569554e+09 _runtime‚2123.706888366 _step‚179 & lr-AdamW/pg1‚0.0007843692624318063 & lr-AdamW/pg2‚0.0007843692624318063 trainer/global_step‚2249*GÆòÌªÆ & lr-AdamW/pg2‚0.0007843692624318063 trainer/global_step‚2249 % _timestamp‚1.7884186529569554e+09 _runtime‚2123.706888366 _step‚179 & lr-AdamW/pg1‚0.0007843692624318063³lJŸ2«- _wandbruntime‚2123  _runtime‚2123dpÊKÇ'¬ÁO YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98609877428736 _step‚180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94373477790720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94373453850624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14924379 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚87400236149767 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98972650 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚87924308955136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚663 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92083654605824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40820873 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884186529578414e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58177447 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4159345650688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚87924290952192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58177447 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40820873 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4147290379643 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4159117448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4159117448192 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚87400476838267 _runtime‚2123.708052639 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58151779 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92083408400384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98610191162880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40820871 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38799131 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38799796 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23874752 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428c†@­ÆO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98610191162880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40820871 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38799131 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38799796 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23874752 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98609877428736 _step‚180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94373477790720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94373453850624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14924379 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚87400236149767 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98972650 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚87924308955136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚663 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92083654605824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40820873 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_ú’kðŒbatch_start/active.small_pool.peak‚31166 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884186529578414e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58177447 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4159345650688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚87924290952192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58177447 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40820873 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4147290379643 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4159117448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4159117448192 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚87400476838267 _runtime‚2123.708052639 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58151779 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92083408400384´±ÆÁ?2®- _wandbruntime‚2124  _runtime‚2124‘“/oÑ&¯ËM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99015920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚87962628896256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚87438611695931 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚87962628896256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99015920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94414647486976 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4148899130103 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4160955142656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚87962646899200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58203024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58177330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277197824 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83289488916480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 trainer/global_step‚2249 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40838592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2124.557180134 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38816303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14930571 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23885107 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98653215890944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 _step‚181 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40838592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99041616 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38815678 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23885109 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92123830257664 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301137920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4161183358464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14931194 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99015920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚87962628896256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚87438611695931 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚87962628896256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99015920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94414647486976 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4148899130103 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4160955142656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚87962646899200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58203024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58177330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277197824 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83289488916480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649348 trainer/global_step‚2249 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40838592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2124.557180134 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38816303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14930571 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23885107 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98653215890944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 _step‚181 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40838592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99041616 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38815678 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23885109 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92123830257664 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301137920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4161183358464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14931194 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚89307233302275 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚85068292312064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4238700301711 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24392504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59432681 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101165172 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2168.809544724 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94092204740608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 trainer/global_step‚2299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59432681 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚94092450946048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59458349 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39642652 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚96431962603520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4330127027712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4329854010880 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100761792674304 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4251244787712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296956928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚85068309351424 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101139502 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41706821 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚652 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚184 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800‰–êÒ'¿ÌO BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94092204740608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 trainer/global_step‚2299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541891584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59432681 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚94092450946048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59458349 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39642652 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚96431962603520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4330127027712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4329854010880 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100761792674304 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4251244787712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25668 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296956928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚85068309351424 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101139502 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41706821 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚652 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚184 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚89307233302275 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚85068292312064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4238700301711 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24392504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59432681 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101165172 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2168.809544724 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚89345368159939 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚85104611809280 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39658501 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚96473132299776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15255644 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4253082495488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4331725395968 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚94132380379136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41724540 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101208468 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41724540 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚89345127471231 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4252854279680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚85104594769920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101208468 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89879526099456 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚94132626597888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚185 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89879544102400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101182772 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 264614912 % _timestamp‚1.7884186989146216e+09 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4252854279680 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 _runtime‚2169.665580111 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59458232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 288555008 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4331990010880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚94132626597888 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚185 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89879544102400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101182772 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 264614912 % _timestamp‚1.7884186989146216e+09 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4252854279680 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 _runtime‚2169.665580111 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59458232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 288555008 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4331990010880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2A—ëkD GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25694 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚94132626597888 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚89345368159939 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚85104611809280 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39658501 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚96473132299776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15255644 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4253082495488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4331725395968 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚94132380379136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41724540 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101208468 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41724540 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚89345127471231 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4252854279680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚85104594769920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101208468 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89879526099456 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚94132626597888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¹Lš2Ã- _wandbruntime‚2169  _runtime‚2169dMîxHÄ 1loss/SRqqt4/detection_loss‚2.0409994109951217 0loss/FRt2/assignment_loss‚2.2545888662338256 2loss/SRqqt2/assignment_loss‚1.8193612688149396 1loss/SRqqt4/assignment_loss‚2.159548183967327 loss/FBt3/assignment_loss‚0 0loss/FRt3/assignment_loss‚3.5435601879910723 1loss/SRqqt2/detection_loss‚1.3156842741828698 _runtime‚2169.666659855 _step‚186 1loss/SRqqt1/assignment_loss‚2.175941633681456 /loss/FRt3/detection_loss‚2.4513502811513295 .loss/FRt4/detection_loss‚2.298406533104308 /loss/FBt3/detection_loss‚2.1049469709396362 .loss/FBt4/detection_loss‚4.435206174850464 %loss/total_loss‚50.07872469573759 1loss/SRqqt3/assignment_loss‚2.231317533105612 0loss/FBt1/assignment_loss‚0.8429775883754095 0loss/FBt4/assignment_loss‚0.5058390969410539 /loss/FRt1/detection_loss‚2.5714064647511736 /loss/FRt2/detection_loss‚1.7543303495893876 .loss/FBt2/detection_loss‚0.546992371853349 epoch‚1 trainer/global_step‚2299 0loss/FRt1/assignment_loss‚3.8960909116558913 0loss/FRt4/assignment_loss‚3.9529250441713537 0loss/FBt2/assignment_loss‚0.2650675385716866 /loss/FBt1/detection_loss‚2.3922394116719565 1loss/SRqqt3/detection_loss‚1.9546233159303665 % _timestamp‚1.7884186989168475e+09 1loss/SRqqt1/detection_loss‚2.5653212832079992=ÆÄàMÅÇ 0loss/FRt3/assignment_loss‚3.5435601879910723 1loss/SRqqt2/detection_loss‚1.3156842741828698 _runtime‚2169.666659855 _step‚186 1loss/SRqqt1/assignment_loss‚2.175941633681456 /loss/FRt3/detection_loss‚2.4513502811513295 .loss/FRt4/detection_loss‚2.298406533104308 /loss/FBt3/detection_loss‚2.1049469709396362 .loss/FBt4/detection_loss‚4.435206174850464 %loss/total_loss‚50.07872469573759 1loss/SRqqt3/assignment_loss‚2.231317533105612 0loss/FBt1/assignment_loss‚0.8429775883754095 0loss/FBt4/assignment_loss‚0.5058390969410539 /loss/FRt1/detection_loss‚2.5714064647511736 /loss/FRt2/detection_loss‚1.7543303495893876 .loss/FBt2/detection_loss‚0.546992371853349 epoch‚1 trainer/global_step‚2299 0loss/FRt1/assignment_loss‚3.8960909116558913 0loss/FRt4/assignment_loss‚3.9529250441713537 0loss/FBt2/assignment_loss‚0.2650675385716866 /loss/FBt1/detection_loss‚2.3922394116719565 1loss/SRqqt3/detection_loss‚1.9546233159303665 % _timestamp‚1.7884186989168475e+09 1loss/SRqqt1/detection_loss‚2.5653212832079992 1loss/SRqqt4/detection_loss‚2.0409994109951217 0loss/FRt2/assignment_loss‚2.2545888662338256 2loss/SRqqt2/assignment_loss‚1.8193612688149396 1loss/SRqqt4/assignment_loss‚2.159548183967327 loss/FBt3/assignment_loss‚0º¥0›Æ‚(: µäÔüûx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Yé?mÇ‚(:â µäÔÜ®Õz disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.out‚0 disk.sda.in‚ 7.23828125 disk.sda3.in‚7.1875" disk.nvme0n1.out‚ 9031.05859375 disk.dm-1.out‚ 173.71484375$ memory_percent‚99.94313716888428 disk.rbd3.out‚0 disk.dm-1.in‚7.1875 disk.rbd2.in‚0! disk.rbd2.out‚106.36376953125 network.sent‚ 4.326248e+06 disk.rbd1.out‚ 0.98291015625 network.recv‚791100 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 173.71484375 disk.sda2.in‚0 disk.rbd0.in‚0 cpu‚25.177868287240138) disk./.usagePercent‚73.60558865880888 disk.rbd4.in‚0$ disk.nvme0c0n1.out‚ 9031.05859375 disk.rbd3.in‚0 disk.dm-0.out‚ 9031.05859375 disk.sda.out‚ 173.75390625 disk.sda2.out‚ 0.0390625% disk./.usageGB‚1096.7764625549316 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2841.1015625* proc.memory.percent‚17.340707778930664 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4.9296875 disk.dm-0.in‚ 4.9296875& proc.memory.availableMB‚ 9.31640625+ §šÈ‚(: ŸµäÔò›†x gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.21' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.241999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò…¹\dÉ‚(:Ù ŸµäÔ¦¢¢{ disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.1875 disk.nvme0n1.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0! disk.rbd2.out‚107.87548828125 disk.sda.in‚ 7.23828125 disk.drbd1777.out‚0 disk.sda1.out‚0% disk./.usageGB‚1097.0066947937012 disk.sda2.in‚0 disk.sda.out‚174.125% proc.memory.availableMB‚ 9.5546875* proc.memory.percent‚17.340707778930664 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.dm-1.out‚ 174.0859375 cpu‚25.17710181740827) disk./.usagePercent‚73.62103973753042 disk.rbd1.out‚ 0.98291015625" disk.nvme0n1.out‚ 9164.67578125 disk.dm-1.in‚7.1875 disk.rbd0.in‚0 disk.rbd3.in‚0$ memory_percent‚99.94168281555176" proc.memory.rssMB‚ 2841.1015625 network.recv‚797685 disk.sda3.out‚ 174.0859375 disk.dm-0.in‚ 4.9296875 disk.drbd1777.in‚ 1.03515625 disk.rbd3.out‚0 network.sent‚ 4.434502e+06 disk.dm-0.out‚ 9164.67578125$ disk.nvme0c0n1.out‚ 9164.67578125 disk.rbd1.in‚0 disk.rbd4.in‚0†ýқʂ(: ®µäÔËÔx gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.080399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ×mË‚(:â ®µäÔóÞ­z disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd3.in‚0 disk.sda3.in‚7.1875 disk.nvme0n1.in‚ 4.9296875% disk.nvme0n1.out‚9165.12744140625" disk.dm-0.out‚9165.12744140625 disk.dm-0.in‚ 4.9296875" proc.memory.rssMB‚ 2841.1015625! disk./.usageGB‚1097.037109375 disk.rbd2.in‚0 disk.sda3.out‚ 174.5703125* proc.memory.percent‚17.340707778930664 disk.sda2.in‚0 disk.dm-1.in‚7.1875 disk.rbd4.in‚0 disk.rbd4.out‚0 proc.cpu.threads‚17 network.sent‚ 4.440327e+06 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.out‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚9165.12744140625 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda.in‚ 7.23828125$ memory_percent‚99.94168281555176 cpu‚25.176156216731137 network.recv‚801652) disk./.usagePercent‚73.62308088560093 disk.dm-1.out‚ 174.5703125! disk.rbd2.out‚108.01611328125 disk.rbd0.out‚0 disk.rbd1.out‚ 0.98291015625 disk.sda2.out‚ 0.0390625 disk.sda.out‚ 174.609375% proc.memory.availableMB‚ 9.5546875c™ot2Ì- _wandbruntime‚2214  _runtime‚2214Ú-fÆÍÀ $ _timestamp‚1.788418743543793e+09 _runtime‚2214.293744702 _step‚187 & lr-AdamW/pg1‚0.0007807841911426455 & lr-AdamW/pg2‚0.0007807841911426455 trainer/global_step‚2349¨X¦ŽËÎÅ & lr-AdamW/pg1‚0.0007807841911426455 & lr-AdamW/pg2‚0.0007807841911426455 trainer/global_step‚2349 $ _timestamp‚1.788418743543793e+09 _runtime‚2214.293744702 _step‚187»îö>2Ï- _wandbruntime‚2214  _runtime‚2214d"šÌ'ÐÆO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4423269062656 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4343143924736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40483554 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96101001080832 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96101247286272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 264628224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103306354 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4330333872919 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42592773 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚91758085358592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _step‚188 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4342915722240 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40484170 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86883415204864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96101001080832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚614 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98490423476224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60713583 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4330557522059 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚91758085358592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60739251 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚5610î¶%ÑËO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 264628224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25670 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103306354 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4330333872919 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42592773 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚91758085358592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _step‚188 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4342915722240 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25668 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40484170 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86883415204864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96101001080832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚614 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98490423476224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31395 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60713583 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4330557522059 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚91758085358592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60739251 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4423269062656 ?9DeviceStatsMonitor.on_train_batch_starˆ—‚Át/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739343872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4343143924736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40483554 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96101001080832 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96101247286272¼8ÍeÊ2Ò- _wandbruntime‚2215  _runtime‚2215 †’È×&ÓÑM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86919734702080 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 _runtime‚2215.181293529 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24920609 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24920607 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4425406155264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4344981632512 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40499933 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103375320 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275100672 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42610490 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60764828 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15579943 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98531593172480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103349624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91251883935239 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2349 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96141422938112 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91252124623947 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4344753416704 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91796441305600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91796423302656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60739134 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103349624 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91796441305600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103375320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299040768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42610492 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824ŽûqÜ&ÔÖM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42610492 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86919734702080 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 _runtime‚2215.181293529 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24920609 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24920607 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4425406155264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4344981632512 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40499933 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103375320 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228215808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275100672 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688708 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42610490 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60764828 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15579943 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98531593172480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103349624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91251883935239 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2349 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96141422938112 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91252124623947 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4344753416704 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91796441305600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91796423302656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60739134 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103349624 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91796441305600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103375320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299040768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299½à Ð2Õ- _wandbruntime‚2215  _runtime‚2215 |xFFÖÀ /loss/FBt4/assignment_loss‚0.590417992323637 2loss/SRqqt2/assignment_loss‚1.1715039588132148 1loss/SRqqt3/assignment_loss‚2.255530124530196 /loss/FRt3/detection_loss‚2.4749701129065618 /loss/FRt2/detection_loss‚1.8126190840094178 1loss/SRqqt2/detection_loss‚1.1783150714837898 /loss/FBt1/detection_loss‚2.3184851706027985 % _timestamp‚1.7884187444327652e+09 /loss/FBt4/detection_loss‚2.4047921191562307 1loss/SRqqt3/detection_loss‚2.0664017972137247 /loss/FBt2/detection_loss‚0.7703540771739719 trainer/global_step‚2349 /loss/FRt4/assignment_loss‚4.461424063804538 loss/FBt3/assignment_loss‚0 /loss/FRt1/detection_loss‚2.4979413144967775 0loss/SRqqt1/detection_loss‚2.427536961105135 /loss/FBt3/detection_loss‚1.3287930488586426 _step‚190 2loss/SRqqt4/assignment_loss‚2.1813925428717744 0loss/SRqqt4/detection_loss‚2.066142107926163 _runtime‚2215.182659572 /loss/FRt2/assignment_loss‚2.181572569741143 /loss/FBt1/assignment_loss‚0.811555320380478 1loss/FBt2/assignment_loss‚0.26375660353161423 -loss/FRt4/detection_loss‚2.44279308374538 &loss/total_loss‚47.749943453202704 epoch‚1 /loss/FRt1/assignment_loss‚4.144192153757269 0loss/FRt3/assignment_loss‚3.9407459100087485 2loss/SRqqt1/assignment_loss‚1.9587082647615008Þ«¬K×Å 2loss/SRqqt4/assignment_loss‚2.1813925428717744 0loss/SRqqt4/detection_loss‚2.066142107926163 _runtime‚2215.182659572 /loss/FRt2/assignment_loss‚2.181572569741143 /loss/FBt1/assignment_loss‚0.811555320380478 1loss/FBt2/assignment_loss‚0.26375660353161423 -loss/FRt4/detection_loss‚2.44279308374538 &loss/total_loss‚47.749943453202704 epoch‚1 /loss/FRt1/assignment_loss‚4.144192153757269 0loss/FRt3/assignment_loss‚3.9407459100087485 2loss/SRqqt1/assignment_loss‚1.9587082647615008 /loss/FBt4/assignment_loss‚0.590417992323637 2loss/SRqqt2/assignment_loss‚1.1715039588132148 1loss/SRqqt3/assignment_loss‚2.255530124530196 /loss/FRt3/detection_loss‚2.4749701129065618 /loss/FRt2/detection_loss‚1.8126190840094178 1loss/SRqqt2/detection_loss‚1.1783150714837898 /loss/FBt1/detection_loss‚2.3184851706027985 % _timestamp‚1.7884187444327652e+09 /loss/FBt4/detection_loss‚2.4047921191562307 1loss/SRqqt3/detection_loss‚2.0664017972137247 /loss/FBt2/detection_loss‚0.7703540771739719 trainer/global_step‚2349 /loss/FRt4/assignment_loss‚4.461424063804538 loss/FBt3/assignment_loss‚0 /loss/FRt1/detection_loss‚2.4979413144967775 0loss/SRqqt1/detection_loss‚2.427536961105135 /loss/FBt3/detection_loss‚1.3287930488586426 _step‚190¾¢Ví ›Ø‚(: ½µäÔÆ©x gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.341' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.868199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ܦ†nÙ‚(:ã ½µäÔù½ú{' disk.nvme0c0n1.out‚9299.23974609375 network.sent‚ 4.446613e+06 disk.dm-1.out‚ 175.0234375( disk./.usagePercent‚73.6230455565612 disk.sda3.in‚7.1875 disk.nvme0n1.in‚ 4.9296875% disk.nvme0n1.out‚9299.23974609375 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda.out‚175.0625 disk.rbd1.in‚0 cpu‚25.175276620564745 disk.rbd2.in‚0 disk.rbd3.out‚0% disk./.usageGB‚1097.0365829467773 disk.sda3.out‚ 175.0234375" disk.dm-0.out‚9299.23974609375! disk.rbd2.out‚108.01611328125 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0 network.recv‚806343 disk.dm-0.in‚ 4.9296875 disk.drbd1777.out‚0$ memory_percent‚99.94311332702637% proc.memory.availableMB‚ 9.3203125" proc.memory.rssMB‚ 2841.1015625 disk.rbd1.out‚ 0.98291015625 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.23828125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.1875 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚ 4.9296875* proc.memory.percent‚17.340707778930664röÇŽÚ‚(:ƒ ̵äÔˆ¯Òx gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Eé'rÛ‚(:ç ̵äÔ¼ÂØz disk.rbd3.in‚0 disk.rbd2.out‚108.1806640625 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚9299.49755859375 disk.rbd1.out‚ 0.98291015625% disk.nvme0n1.out‚9299.49755859375" proc.memory.rssMB‚ 2841.1015625) disk./.usagePercent‚73.62304965268174% disk./.usageGB‚1097.0366439819336 disk.dm-0.in‚ 4.9296875 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0* proc.memory.percent‚17.340707778930664 disk.nvme0c0n1.in‚ 4.9296875 disk.sda.in‚ 7.23828125 disk.sda3.out‚ 175.72265625 disk.rbd1.in‚0" disk.dm-0.out‚9299.49755859375 disk.drbd1777.in‚ 1.03515625$ proc.memory.availableMB‚9.515625 disk.rbd3.out‚0 disk.rbd4.in‚0$ memory_percent‚99.94192123413086 network.sent‚ 4.553994e+06 disk.sda.out‚ 175.76171875 disk.sda3.in‚7.1875 disk.nvme0n1.in‚ 4.9296875 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.1875 disk.rbd2.in‚0 network.recv‚811593 disk.sda1.in‚ 0.05078125 disk.dm-1.out‚ 175.72265625 disk.rbd0.out‚0 disk.rbd4.out‚0 cpu‚25.17436442298678<åuŽÜ‚(:ƒ ÛµäÔ˜Îòy gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»d2rÝ‚(:ç ÛµäÔ¾®ë{" disk.dm-0.out‚9299.93798828125$ memory_percent‚99.94192123413086 disk.rbd4.out‚0' disk.nvme0c0n1.out‚9299.93798828125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4.9296875 disk.nvme0c0n1.in‚ 4.9296875* proc.memory.percent‚17.340707778930664 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 7.23828125 disk.sda.out‚ 176.12890625 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2841.1015625 network.sent‚ 4.560123e+06 disk.sda3.in‚7.1875 disk.rbd1.out‚ 0.98291015625% disk.nvme0n1.out‚9299.93798828125 network.recv‚815877 disk.sda3.out‚ 176.08984375 disk.rbd1.in‚0 disk.dm-1.in‚7.1875 disk.sda1.in‚ 0.05078125 cpu‚25.173628224836822$ disk./.usageGB‚1097.036766052246 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 176.08984375 disk.rbd2.out‚108.1806640625 disk.drbd1777.out‚0$ proc.memory.availableMB‚9.515625) disk./.usagePercent‚73.62305784492284 disk.dm-0.in‚ 4.9296875Ö¦d—2Þ- _wandbruntime‚2259  _runtime‚2259eB}ÇßÁ _step‚191 & lr-AdamW/pg1‚0.0007788727946991576 & lr-AdamW/pg2‚0.0007788727946991576 trainer/global_step‚2399 % _timestamp‚1.7884187892467651e+09 _runtime‚2259.996706892€Ã‹ÀÌàÆ trainer/global_step‚2399 % _timestamp‚1.7884187892467651e+09 _runtime‚2259.996706892 _step‚191 & lr-AdamW/pg1‚0.0007788727946991576 & lr-AdamW/pg2‚0.0007788727946991576¿õhX^2á- _wandbruntime‚2259  _runtime‚2259|¯@ø¿âÈO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4434814859264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270919680 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62020153 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚93675000564736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25428004 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚652 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚88698555137024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61994485 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105498876 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105065892126208 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4434814859264 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62020153 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43478723 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚93674982561792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61994485 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98109797421056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15898052 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15897400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98109797421056 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4435043061760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25428002 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294859776 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚93675000564736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43478721 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚93674982561792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43478723 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105065597266432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2399 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440)°—ÈÓ'ãÍO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚93674982561792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541891584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43478723 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105065597266432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2399 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4434814859264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270919680 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62020153 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚93675000564736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25428004 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31395 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚652 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688500 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528065 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚88698555137024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61994485 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105498876 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105065892126208 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228202496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4434814859264 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62020153 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43478723 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31166 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739343872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31166 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚93674982561792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61994485 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98109797421056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15898052 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15897400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98109797421056 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4435043061760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228202496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25428002 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294859776 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚93675000564736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25670 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294665 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43478721 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096Àu P2ä- _wandbruntime‚2260  _runtime‚2260¡V¼Ù&åÓM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚93713320505856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚93713320505856 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105108630342144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4436652553728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105542172 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚88734857594880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4436652553728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25438357 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚93158881087955 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚88734840555520 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41342035 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105542172 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105516476 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105108933577216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100590077985280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4518552356864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98150219278336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4436880769536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100590101925376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303235072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 _step‚193 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43496442 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚93158640399247 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43496440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚93713338508800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31395 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚2399 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105516476 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62020036 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992,í.­ðæØM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚93158881087955 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚88734840555520 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41342035 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541891584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105542172 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105516476 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105108933577216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739343872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100590077985280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4518552356864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98150219278336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4436880769536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100590101925376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303235072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528065 _step‚193 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43496442 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246218752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31166 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚93158640399247 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43496440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚93713338508800 C9DeviceStatsMonitor.on_train_batch_end/aÓÂÀAî llocation.all.peak‚31395 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚2399 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228215808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105516476 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31395 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62020036 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246218752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚93713320505856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25694 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739343872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294665 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚93713320505856 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105108630342144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541891584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4436652553728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105542172 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚88734857594880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4436652553728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25438357Ám92ç- _wandbruntime‚2260  _runtime‚226031WèÑ 0loss/FRt1/assignment_loss‚4.1480204745342855 /loss/FRt3/assignment_loss‚4.180693998569396 0loss/FRt4/assignment_loss‚3.7889036660970645 1loss/SRqqt2/detection_loss‚1.1834474517635896 /loss/FBt1/detection_loss‚1.6585439682006835 % _timestamp‚1.7884187901140308e+09 0loss/SRqqt1/detection_loss‚2.726529345367894 $loss/total_loss‚49.8383064912329 .loss/FBt4/detection_loss‚2.351882517337799 0loss/FBt1/assignment_loss‚0.7816612283388774 0loss/FBt3/assignment_loss‚0.6359481502925822 0loss/FBt4/assignment_loss‚0.8522209576331079 1loss/FBt2/assignment_loss‚0.18381573777459068 -loss/FRt3/detection_loss‚2.55414778866419 /loss/FRt4/detection_loss‚2.3389883138412655 /loss/FBt3/detection_loss‚3.9150914748509726 _step‚194 1loss/SRqqt1/assignment_loss‚1.992919770154086 0loss/SRqqt4/detection_loss‚1.909461689091498 /loss/FBt2/detection_loss‚0.6548211370430449 1loss/SRqqt2/assignment_loss‚1.510229877183135 2loss/SRqqt3/assignment_loss‚1.9567097239988915 2loss/SRqqt4/assignment_loss‚2.0990212857723236 /loss/FRt1/detection_loss‚2.5259363714017367 /loss/FRt2/detection_loss‚1.8815708619356155 1loss/SRqqt3/detection_loss‚1.9033791017025075 0loss/FRt2/assignment_loss‚2.1043615996837617 epoch‚1 trainer/global_step‚2399 _runtime‚2260.863863073XûT©\éÖ 0loss/FRt1/assignment_loss‚4.1480204745342855 /loss/FRt3/assignment_loss‚4.180693998569396 0loss/FRt4/assignment_loss‚3.7889036660970645 1loss/SRqqt2/detection_loss‚1.1834474517635896 /loss/FBt1/detection_loss‚1.6585439682006835 % _timestamp‚1.7884187901140308e+09 0loss/SRqqt1/detection_loss‚2.726529345367894 $loss/total_loss‚49.8383064912329 .loss/FBt4/detection_loss‚2.351882517337799 0loss/FBt1/assignment_loss‚0.7816612283388774 0loss/FBt3/assignment_loss‚0.6359481502925822 0loss/FBt4/assignment_loss‚0.8522209576331079 1loss/FBt2/assignment_loss‚0.18381573777459068 -loss/FRt3/detection_loss‚2.55414778866419 /loss/FRt4/detection_loss‚2.3389883138412655 /loss/FBt3/detection_loss‚3.9150914748509726 _step‚194 1loss/SRqqt1/assignment_loss‚1.992919770154086 0loss/SRqqt4/detection_loss‚1.909461689091498 /loss/FBt2/detection_loss‚0.6548211370430449 1loss/SRqqt2/assignment_loss‚1.510229877183135 2loss/SRqqt3/assignment_loss‚1.9567097239988915 2loss/SRqqt4/assignment_loss‚2.0990212857723236 /loss/FRt1/detection_loss‚2.5259363714017367 /loss/FRt2/detection_loss‚1.8815708619356155 1loss/SRqqt3/detection_loss‚1.9033791017025075 0loss/FRt2/assignment_loss‚2.1043615996837617 epoch‚1 trainer/global_step‚2399 _runtime‚2260.863863073‹ÿ4¾›ê‚(: êµäÔµ ˆx gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.467' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.293400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±\ ©lë‚(:á êµäÔ”ˆíz disk.rbd1.in‚0# proc.memory.availableMB‚9.28125 network.sent‚ 4.56594e+06 disk.dm-1.out‚ 176.6640625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 176.6640625 proc.cpu.threads‚17 disk.dm-0.in‚ 4.9296875 disk.dm-1.in‚7.1875 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚9308.69482421875 disk.sda1.out‚0 network.recv‚819864 disk.rbd2.out‚109.7705078125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda.out‚ 176.703125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd3.out‚0 disk.rbd1.out‚ 0.98291015625% disk.nvme0n1.out‚9308.69482421875" proc.memory.rssMB‚ 2841.1015625 disk.sda.in‚ 7.23828125 disk.nvme0n1.in‚ 4.9296875* proc.memory.percent‚17.340707778930664) disk./.usagePercent‚73.62361389328721$ disk./.usageGB‚1097.045051574707 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.sda3.in‚7.1875$ memory_percent‚99.94335174560547 cpu‚25.172732579521455" disk.dm-0.out‚9308.69482421875 disk.sda2.out‚ 0.0390625⸚ýì‚(:… ùµäÔ“ÐÇx gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚124.576' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.9152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µgÞˆtí‚(:é ùµäÔ¼·ê{ disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 0.98291015625% disk.nvme0n1.out‚9309.57080078125* proc.memory.percent‚17.384028434753418 disk.rbd0.out‚0 disk.sda3.in‚7.1875 disk.sda3.out‚ 177.03515625 disk.nvme0n1.in‚ 4.9296875 network.sent‚ 4.674647e+06) disk./.usagePercent‚73.62361389328721 disk.dm-1.in‚7.1875 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 177.03515625 disk.rbd4.out‚0" disk.dm-0.out‚9309.57080078125 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚ 177.07421875% proc.memory.availableMB‚ 6.2578125$ disk./.usageGB‚1097.045051574707 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.in‚ 7.23828125 network.recv‚826708 disk.sda2.in‚0 disk.rbd2.out‚109.7705078125 disk.rbd4.in‚0$ memory_percent‚99.96180534362793# proc.memory.rssMB‚ 2848.19921875 cpu‚25.176572799433735 disk.dm-0.in‚ 4.9296875' disk.nvme0c0n1.out‚9309.57080078125 disk.sda1.in‚ 0.05078125šë{Å›î‚(: ˆ¶äÔ܉˜x gpu.0.gpu‚36.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.969' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.593799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cðßrï‚(:ç ˆ¶äÔû÷¬z disk.dm-0.in‚ 4.9296875" proc.memory.rssMB‚ 2859.3984375 disk.dm-1.in‚7.1875! disk.rbd2.out‚109.93505859375 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.out‚0$ disk.nvme0n1.out‚9310.1552734375 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.sda3.out‚ 177.73828125$ memory_percent‚99.97665882110596* proc.memory.percent‚17.452383041381836 cpu‚25.206588013135747 network.sent‚ 4.679998e+06 network.recv‚829920) disk./.usagePercent‚73.62361005317419 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.nvme0n1.in‚ 4.9296875 disk.rbd3.in‚0 disk.sda.out‚ 177.77734375 disk.rbd1.out‚ 0.98291015625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd4.out‚0& disk.nvme0c0n1.out‚9310.1552734375 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda.in‚ 7.23828125 disk.sda3.in‚7.1875! disk.dm-0.out‚9310.1552734375$ disk./.usageGB‚1097.044994354248 disk.dm-1.out‚ 177.73828125 disk.drbd1777.out‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 3.82421875 disk.rbd2.in‚0{EŽð‚(:ƒ —¶äÔåêËy gpu.0.gpu‚46.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08,vqñ‚(:æ —¶äÔ“¾Âz disk.nvme0c0n1.in‚ 4.9296875 disk.sda.out‚ 178.203125) proc.memory.percent‚17.43922233581543% disk./.usageGB‚1097.1699333190918' disk.nvme0c0n1.out‚9409.52099609375 disk.sda3.out‚ 178.1640625 disk.rbd1.out‚ 0.98291015625 disk.dm-1.out‚ 178.1640625! disk.rbd2.out‚109.93505859375 proc.cpu.threads‚17 network.sent‚ 4.686015e+06 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0& proc.memory.availableMB‚ 5.22265625 cpu‚25.236715618282428 network.recv‚834548) disk./.usagePercent‚73.63199481193564 disk.drbd1777.out‚0 disk.rbd1.in‚0" disk.dm-0.out‚9409.52099609375 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.23828125 disk.dm-0.in‚ 4.9296875$ memory_percent‚99.96812343597412 disk.rbd3.out‚0 disk.sda3.in‚7.1875 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.9296875% disk.nvme0n1.out‚9409.52099609375 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.1875 disk.rbd0.in‚0" proc.memory.rssMB‚ 2857.2421875 disk.rbd4.out‚0;3Þžšò‚(: ¦¶äÔ‚›šx gpu.0.gpu‚46.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.648' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.729599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýèHŒjó‚(:ß ¦¶äÔ˜ÐÈz) disk./.usagePercent‚73.64038366681763 disk.sda3.out‚178.8125 disk.sda2.in‚0 disk.sda.out‚ 178.8515625% disk./.usageGB‚1097.2949333190918 disk.rbd1.out‚ 0.98291015625 disk.dm-1.in‚7.1875 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.9296875 disk.dm-1.out‚178.8125$ memory_percent‚99.97594356536865 disk.rbd4.in‚0 disk.sda.in‚ 7.23828125% disk.nvme0n1.out‚9508.63818359375 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 cpu‚25.264441601549727 disk.rbd4.out‚0 disk.sda3.in‚7.1875 disk.dm-0.in‚ 4.9296875 network.sent‚ 4.691885e+06' disk.nvme0c0n1.out‚9508.63818359375 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0 network.recv‚838567 disk.rbd3.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚110.07177734375& proc.memory.availableMB‚ 3.94140625 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.in‚ 0.05078125" disk.dm-0.out‚9508.63818359375 proc.memory.rssMB‚ 2866.71875) proc.memory.percent‚17.49706268310547|3)2ô- _wandbruntime‚2329  _runtime‚2329VD¯P:*õ´T LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚299 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚100173353907712 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 184860752 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6739343872 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚95077389159791 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 549453824 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3428 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚42354194 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6522528065 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4489051710464 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚44547246 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 trainer/global_step‚2417 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6252512256 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6739343872 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚44547248 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚107246281298944 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚107246095621632 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚95077389159791 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 549453824 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3428 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚42354194 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6522528065 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4489051710464 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚44547246 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 trainer/global_step‚2417Ç£8d2÷- _wandbruntime‚2330  _runtime‚2330¼üÿ¸D)ø¾R 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31395 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚2417 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚100199887492608 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 549453824 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31395 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚95710460989952 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 4489616215552 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚26072949 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 \ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚90898432 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6252512256 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31166 _step‚196 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚62919292 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚16295781 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4572183034368 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4572021297152 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚299 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚102701906051584 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31166 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4489444505600 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20199 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚16295408 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6343884800 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 107481476 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 171709952 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884188598050144e+09 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚44562184 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚44562184 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚44562182 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚95102348425183 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4476416165855 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚561 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚375 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1261 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 6893338624 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20199 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 537294665 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3428 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚95710460989952 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚95710442987008 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337968128 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚90625932259328 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚42368355 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚26072947 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6522528065 _runtime‚2330.555313418 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚100200077205504 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6739343872 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚373 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚107274089085952 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 189712896 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20201 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚62919292 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚100200077205504 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 541891584 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚262 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚62899093 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20201 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3428 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚62899093 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 167821392 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚100199887492608 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚90625949298688 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 161737216 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚299 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚44562182 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚23 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 6893338624 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 107461275 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 189712896 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 541891584 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚262 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6739343872 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚561 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚262 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31395 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚2417 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚100199887492608 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 549453824 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31395 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚95710460989952 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 4489616215552 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚26072949 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 \ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚90898432 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6252512256 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31166 _step‚196 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚62919292 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚16295781 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4572183034368 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4572021297152 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚299 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚102701906051584 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31166 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4489444505600 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20199 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚16295408 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6343884800 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 107481476 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 171709952 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884188598050144e+09 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚44562184 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚44562184 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚44562182 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚95102348425183 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4476416165855 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚561 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚375 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1261 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 6893338624 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20199 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 537294665 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3428 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚95710460989952 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚95710442987008 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337968128 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚90625932259328 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚42368355 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚26072947 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6522528065 _runtime‚2330.555313418 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚100200077205504 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6739343872 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚373 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚107274089085952 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 189712896 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20201 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚62919292 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚100200077205504 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 541891584 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚262 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚62899093 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20201 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3428 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚62899093 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 167821392 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚100199887492608 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚90625949298688 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944Äù|î>ú‚(:„ µ¶äÔñ·šy gpu.0.gpu‚45.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.248' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\Q¬öpû‚(:å µ¶äÔÀÞ…{" disk.dm-0.out‚9509.19189453125 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 179.15625' disk.nvme0c0n1.out‚9509.19189453125 network.recv‚843430 disk.sda3.out‚ 179.1171875% disk.nvme0n1.out‚9509.19189453125 disk.dm-1.out‚ 179.1171875' proc.memory.availableMB‚ 12.94921875) proc.memory.percent‚17.43922233581543 disk.rbd3.in‚0 disk.drbd1777.in‚ 1.03515625 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0) disk./.usagePercent‚73.65721053002817 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9296875 disk.dm-0.in‚ 4.9296875 disk.nvme0c0n1.in‚ 4.9296875 network.sent‚ 4.698275e+06 disk.rbd3.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2857.2421875 disk.dm-1.in‚7.1875 disk.rbd2.in‚0 disk.sda.in‚ 7.23828125 disk.sda3.in‚7.1875! disk.rbd2.out‚110.07177734375 disk.drbd1777.out‚0 disk.rbd4.out‚0$ memory_percent‚99.92096424102783 cpu‚25.29339118861653 disk.rbd0.in‚0% disk./.usageGB‚1097.5456657409668 disk.sda1.out‚0 disk.rbd1.out‚ 1.068847656252†`à2ü- _wandbruntime‚2344  _runtime‚2344àþÚcÈý‚¦ ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.14397133886814117 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3038238286972046 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5833333730697632 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.021383099257946014 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.29189184308052063 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9741702675819397 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.42867517471313477 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9087302684783936 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2862076759338379 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1480388343334198 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.008831450715661049 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4027778208255768 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2734006643295288 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.11376003921031952 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3579670488834381 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.29404762387275696 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11141345649957657 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06382978707551956 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.38678160309791565 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.23795118927955627 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.24616047739982605 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39335474371910095 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3114035129547119 +jet/accuracy_3_of_5‚0.04184588044881821 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.009856631979346275 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09674329310655594 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.7953334450721741 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.10016288608312607 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.09793642163276672 6 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.998236358165741 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14805686473846436 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10201061517000198 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.003968254197388887 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_4‚0.2474256455898285 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9926692247390747 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2614942491054535 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.990813672542572 6 Purity/לßÞAw1FRt0SRqqt1FBt/FBt_purity‚0.992137610912323 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10572546720504761 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3194444477558136 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_7‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.29610198736190796 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2586090862751007 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.28816908597946167 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0 jet/accuracy_3_of_7‚0 particle/accuracy_4_of_7‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0833333358168602 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9920635223388672 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_7_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 epoch‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.05736299976706505 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.09793642163276672 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9653022885322571 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3139796257019043 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5681818127632141 particle/accuracy_2_of_2‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.615624189376831 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9629629850387573 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3589441776275635 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.10627340525388718 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929264783859253 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.10040716081857681 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2192022204399109 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9562979936599731 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10244192183017731 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.956250011920929 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11351543664932251 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2142857164144516 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9723795056343079 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1825435608625412 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9818094968795776 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.981903612613678 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.31147849559783936 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8958333134651184 jet/accuracy_6_of_6‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9814444780349731 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.004251700360327959 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.006289308425039053 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.002017068909481168 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12326395511627197 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.41577383875846863 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2944052219390869 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.41087570786476135 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 particle/accuracy_2_of_3‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3117816150188446 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.24450093507766724 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.05828533321619034 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 5 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.99210524559021 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2836601138114929 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.40586280822753906 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_3_of_7‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.29151788353919983 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1419753134250641 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.17651137709617615 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424242198467255 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0635376051068306 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833134651184 particle/accuracy_5_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666865348816 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.27991756796836853 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6287067532539368 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.280303031206131 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.24609753489494324 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.04415504261851311 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9625843167304993 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9758132100105286 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 trainer/global_step‚2417 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9687397480010986 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.10833334177732468 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.327160507440567 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9113475680351257 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09941493719816208 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9819364547729492 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14242424070835114 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9799623489379883 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629629850387573 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.27713194489479065 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11200958490371704 particle/accuracy_3_of_3‚1 +particle/sensitivity‚0.1943599432706833 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9808595776557922 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚0.5 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.45457106828689575 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.228642538189888 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9806202054023743 ,jet/accuracy_3_of_3‚0.009837633930146694 particle/accuracy_2_of_5‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.41608795523643494 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.00511371111497283 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2288617193698883 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.25659170746803284 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2750000059604645 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666865348816 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9784733057022095 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3326471447944641 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4059506058692932 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_6‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1632183939218521 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1339411437511444 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 jet/accuracy_4_of_5‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2626027762889862 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3216294050216675 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.995781421661377 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 *jet/accuracy_1_of_4‚0.7974835634231567 particle/accuracy_3_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9615384340286255 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1024647057056427 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.28404369950294495 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.005610425490885973 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.990384578704834 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.31333333253860474 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2827146649360657 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.27161353826522827 +particle/specificity‚0.9418318271636963 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.19860272109508514 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.28131377696990967 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 +jet/accuracy_3_of_6‚0.03846153989434242 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2790941596031189 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.006465517450124025 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9907464981079102 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10190681368112564 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2669753134250641 jet/accuracy_5_of_6‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 +jet/accuracy_2_of_3‚0.14679653942584991 ,jet/accuracy_3_of_4‚0.021022988483309746 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.08207336068153381 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3606564700603485 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.24814818799495697 particle/accuracy_6_of_6‚1 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 )loss/val_total_loss‚51.56394203742552 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.05941314622759819 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2589668929576874 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.34210526943206787 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9889702200889587 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2846638560295105 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.004870129749178886 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.14506369829177856 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10185185074806213 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 _runtime‚2344.609298721 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2991597354412079 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15798501670360565 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9817543029785156 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9880044460296631 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15988527238368988 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18586885929107666 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10423953086137772 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1073979064822197 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15537890791893005 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2951388955116272 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2128276526927948 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1201949417591095 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012210012646391988 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.27436357736587524 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.41360729932785034 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.005680562928318977 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3371528089046478 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10623165220022202 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.995781421661377 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3221072554588318 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3008376955986023 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.007594651076942682 (particle/accuracy‚0.8251074552536011 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.26450347900390625 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.8893753290176392 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2740740180015564 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3166666626930237 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9947726726531982 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10218039900064468 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9755399823188782 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,jet/accuracy_4_of_4‚0.001873897621408105 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_3‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.39743587374687195 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1680918037891388 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 _step‚197 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7229561805725098 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 % _timestamp‚1.7884188738570948e+09 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2623889744281769 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4328681230545044 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2788257300853729 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10022589564323425 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_4‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.10000448673963547 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4736842215061188 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10350876301527023 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2847222089767456 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.30787989497184753 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.36691561341285706 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.36000001430511475 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.27713194489479065 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 jet/accuracy_4_of_6‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.14391891658306122 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3952829837799072 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.010416666977107525 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0474262535572052 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.15184754133224487 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9403735399246216 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.043434321880340576 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_5‚0.3133064806461334 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_3_of_4‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.1990121603012085 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.5968575477600098 (particle/f_score‚0.25751543045043945 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 jet/accuracy_5_of_7‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.2848837077617645 particle/accuracy_1_of_1‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2873799800872803 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3279891014099121 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 particle/accuracy_4_of_5‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.06444444507360458 jet/accuracy_4_of_7‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.29013749957084656 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.017953671514987946 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.40975654125213623 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 particle/accuracy_4_of_6‚1 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.410194993019104 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_7_of_7‚0.5 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4163203239440918 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9881370067596436 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.30401724576950073 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.21574074029922485 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.2845543920993805 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4178982079029083 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3972051739692688 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4372105300426483 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2597883641719818 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3633072078227997 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14894084632396698 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/3FRt2SRqqt0FBt/event_purity‚0øía±þ‡¦ ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11141345649957657 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06382978707551956 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.38678160309791565 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.23795118927955627 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.24616047739982605 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39335474371910095 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3114035129547119 +jet/accuracy_3_of_5‚0.04184588044881821 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.009856631979346275 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09674329310655ˆ~ê§ù594 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.7953334450721741 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.10016288608312607 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.09793642163276672 6 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.998236358165741 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14805686473846436 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10201061517000198 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.003968254197388887 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_4‚0.2474256455898285 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9926692247390747 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2614942491054535 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.990813672542572 6 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.992137610912323 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10572546720504761 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3194444477558136 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_7‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.29610198736190796 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2586090862751007 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.28816908597946167 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0 jet/accuracy_3_of_7‚0 particle/accuracy_4_of_7‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0833333358168602 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9920635223388672 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_7_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 epoch‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.05736299976706505 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.09793642163276672 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9653022885322571 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3139796257019043 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5681818127632141 particle/accuracy_2_of_2‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.615624189376831 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9629629850387573 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3589441776275635 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.10627340525388718 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929264783859253 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.10040716081857681 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2192022204399109 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9562979936599731 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10244192183017731 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.956250011920929 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11351543664932251 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2142857164144516 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9723795056343079 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1825435608625412 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9818094968795776 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.981903612613678 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.31147849559783936 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8958333134651184 jet/accuracy_6_of_6‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9814444780349731 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.004251700360327959 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.006289308425039053 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.002017068909481168 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12326395511627197 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.41577383875846863 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2944052219390869 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.41087570786476135 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 particle/accuracy_2_of_3‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3117816150188446 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.24450093507766724 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.05828533321619034 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 5 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.99210524559021 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2836601138114929 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.40586280822753906 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_3_of_7‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.29151788353919983 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1419753134250641 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.17651137709617615 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424242198467255 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0635376051068306 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833134651184 particle/accuracy_5_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666865348816 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.27991756796836853 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6287067532539368 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.280303031206131 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.24609753489494324 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.04415504261851311 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9625843167304993 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9758132100105286 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 trainer/global_step‚2417 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9687397480010986 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.10833334177732468 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.327160507440567 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9113475680351257 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09941493719816208 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9819364547729492 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14242424070835114 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9799623489379883 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629629850387573 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.27713194489479065 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11200958490371704 particle/accuracy_3_of_3‚1 +particle/sensitivity‚0.1943599432706833 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9808595776557922 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚0.5 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.45457106828689575 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.228642538189888 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9806202054023743 ,jet/accuracy_3_of_3‚0.009837633930146694 particle/accuracy_2_of_5‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.41608795523643494 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.00511371111497283 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2288617193698883 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.25659170746803284 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2750000059604645 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666865348816 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9784733057022095 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3326471447944641 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4059506058692932 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_6‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1632183939218521 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1339411437511444 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 jet/accuracy_4_of_5‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2626027762889862 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3216294050216675 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.995781421661377 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 *jet/accuracy_1_of_4‚0.7974835634231567 particle/accuracy_3_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9615384340286255 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1024647057056427 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.28404369950294495 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.005610425490885973 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.990384578704834 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.31333333253860474 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2827146649360657 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.27161353826522827 +particle/specificity‚0.9418318271636963 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.19860272109508514 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.28131377696990967 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 +jet/accuracy_3_of_6‚0.03846153989434242 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2790941596031189 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.006465517450124025 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9907464981079102 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10190681368112564 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2669753134250641 jet/accuracy_5_of_6‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 +jet/accuracy_2_of_3‚0.14679653942584991 ,jet/accuracy_3_of_4‚0.021022988483309746 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.08207336068153381 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3606564700603485 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.24814818799495697 particle/accuracy_6_of_6‚1 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 )loss/val_total_loss‚51.56394203742552 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.05941314622759819 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2589668929576874 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.34210526943206787 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9889702200889587 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2846638560295105 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.004870129749178886 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.14506369829177856 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10185185074806213 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 _runtime‚2344.609298721 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2991597354412079 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15798501670360565 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9817543029785156 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9880044460296631 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15988527238368988 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18586885929107666 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10423953086137772 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1073979064822197 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15537890791893005 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2951388955116272 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2128276526927948 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1201949417591095 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012210012646391988 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.27436357736587524 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.41360729932785034 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.005680562928318977 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3371528089046478 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10623165220022202 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.995781421661377 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3221072554588318 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3008376955986023 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.007594651076942682 (particle/accuracy‚0.8251074552536011 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.26450347900390625 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.8893753290176392 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2740740180015564 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3166666626930237 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9947726726531982 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10218039900064468 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9755399823188782 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,jet/accuracy_4_of_4‚0.001873897621408105 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_3‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.39743587374687195 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1680918037891388 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 _step‚197 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7229561805725098 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 % _timestamp‚1.7884188738570948e+09 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2623889744281769 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4328681230545044 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2788257300853729 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10022589564323425 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_4‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.10000448673963547 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4736842215061188 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10350876301527023 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2847222089767456 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.30787989497184753 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.36691561341285706 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.36000001430511475 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.27713194489479065 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 jet/accuracy_4_of_6‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.14391891658306122 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3952829837799072 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.010416666977107525 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0474262535572052 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.15184754133224487 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9403735399246216 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.043434321880340576 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_5‚0.3133064806461334 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_3_of_4‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.1990121603012085 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.5968575477600098 (particle/f_score‚0.25751543045043945 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 jet/accuracy_5_of_7‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.2848837077617645 particle/accuracy_1_of_1‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2873799800872803 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3279891014099121 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 particle/accuracy_4_of_5‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.06444444507360458 jet/accuracy_4_of_7‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.29013749957084656 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.017953671514987946 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.40975654125213623 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 particle/accuracy_4_of_6‚1 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.410194993019104 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_7_of_7‚0.5 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4163203239440918 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9881370067596436 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.30401724576950073 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.21574074029922485 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.2845543920993805 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4178982079029083 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3972051739692688 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4372105300426483 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2597883641719818 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3633072078227997 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14894084632396698 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.14397133886814117 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3038238286972046 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5833333730697632 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_propo²à/d rtion‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.021383099257946014 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.29189184308052063 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9741702675819397 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.42867517471313477 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9087302684783936 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2862076759338379 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1480388343334198 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.008831450715661049 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4027778208255768 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2734006643295288 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.11376003921031952 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3579670488834381 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.29404762387275696 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1ÅlY…ƒÿ‚(:… ĶäÔŠ±Éz gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üdEÕn€‚(:ã ĶäÔ‰Çöz disk.sda2.in‚0 disk.dm-1.out‚179.6875 disk.sda1.out‚0 disk.rbd3.out‚0! disk.rbd2.out‚111.66162109375 network.recv‚848384 disk.nvme0n1.in‚ 4.9296875' disk.nvme0c0n1.out‚9774.94970703125 disk.sda3.out‚179.6875( proc.memory.availableMB‚ 341.66015625" disk.dm-0.out‚9774.94970703125 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.13134765625% disk.nvme0n1.out‚9774.94970703125 disk.sda1.in‚ 0.05078125 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda3.in‚7.1875 disk.rbd1.in‚0) proc.memory.percent‚17.47605800628662 disk.sda.out‚ 179.7265625 network.sent‚ 4.805534e+06% disk./.usageGB‚1097.7080154418945 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.1875 disk.rbd4.out‚0 disk.sda.in‚ 7.23828125$ memory_percent‚97.91467189788818 cpu‚25.300678907590704 proc.cpu.threads‚17( disk./.usagePercent‚73.6681059546779 disk.dm-0.in‚ 4.9296875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2863.27734375ž~ÈÏ‚(:… Ó¶äÔŽ‰x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔÊÀÙp‚‚(:å Ó¶äÔ©ªz disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd1.in‚0$ memory_percent‚97.91338443756104 cpu‚25.29912298653996% disk./.usageGB‚1097.7078666687012 disk.sda2.in‚0 disk.dm-1.in‚7.1875 disk.dm-1.out‚ 180.28125 disk.rbd1.out‚ 1.13134765625 network.sent‚ 4.929456e+06 disk.rbd4.out‚0' disk.nvme0c0n1.out‚9886.10302734375% disk.nvme0n1.out‚9886.10302734375# proc.memory.rssMB‚ 2863.27734375 proc.cpu.threads‚17 disk.sda3.out‚ 180.28125) disk./.usagePercent‚73.66809597038406 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚ 180.3203125 network.recv‚854540 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.sda.in‚ 7.23828125 disk.sda3.in‚7.1875" disk.dm-0.out‚9886.10302734375 disk.rbd3.in‚0 disk.nvme0n1.in‚ 4.9296875! disk.rbd2.out‚111.66162109375 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd2.in‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 341.87109375) proc.memory.percent‚17.47605800628662zro2ƒ- _wandbruntime‚2376  _runtime‚2376l9Ä„¾ % lr-AdamW/pg1‚0.000776882683336909 % lr-AdamW/pg2‚0.000776882683336909 trainer/global_step‚2449 $ _timestamp‚1.788418905650113e+09 _runtime‚2376.400052607 _step‚198Æè¢É…à % lr-AdamW/pg1‚0.000776882683336909 % lr-AdamW/pg2‚0.000776882683336909 trainer/global_step‚2449 $ _timestamp‚1.788418905650113e+09 _runtime‚2376.400052607 _step‚198Ææ€Ù°2†- _wandbruntime‚2376  _runtime‚2376Öö›ŸÒ'‡ÌO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4551477078016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45305639 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101790226597888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97238749519872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63824376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚92073113832448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45305641 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104334561529856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101790226597888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104334537589760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101790472803840 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43076180 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108969883707392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4538485916297 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚101790472803840 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4635322177536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16565694 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4551477078016 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _step‚199 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4538262267153 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 trainer/global_step‚2449 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97238749519872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97238767522816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43075546 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63798707 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4551705281024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63824376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0æ@6×'ˆÑO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104334561529856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101790226597888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104334537589760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101790472803840 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43076180 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108969883707392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4538485916297 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚101790472803840 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4635322177536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16565694 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4551477078016 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _step‚199 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4538262267153 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 trainer/global_step‚2449 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97238749519872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97238767522816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43075546 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63798707 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4551705281024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63824376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4551477078016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45305639 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101790226597888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97238749519872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63824376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚92073113832448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45305641 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824ÇÓ'42‰- _wandbruntime‚2377  _runtime‚23770¡XîwŠ×M 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63824258 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45323358 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16571595 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884189065022385e+09 _runtime‚2377.252995481 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚101830402236416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚109012916813312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96649493917697 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26520839 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109147616 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4553314772480 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109173313 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚101830648455680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109147616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26520841 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63849953 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43092434 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45323360 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97277087463936 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96649734606409 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4637185587200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚92109416290304 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 trainer/global_step‚2449 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97277105466880 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43093072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4553542988800 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109173313 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4553542988800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63849953 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45323360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4636914681344€ƒ â&‹ÜM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4637185587200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚92109416290304 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 trainer/global_step‚2449 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97277105466880 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43093072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4553542988800 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109173313 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4553542988800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63849953 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45323360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4636914681344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63824258 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45323358 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16571595 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884189065022385e+09 _runtime‚2377.252995481 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚101830402236416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚109012916813312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96649493917697 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26520839 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109147616 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4553314772480 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109173313 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚101830648455680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109147616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26520841 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63849953 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43092434 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45323360 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97277087463936 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96649734606409ÈO¾ŸÆ2Œ- _wandbruntime‚2377  _runtime‚2377¬gTÎ /loss/FBt2/detection_loss‚0.7040058941645331 % _timestamp‚1.7884189065042598e+09 1loss/FBt2/assignment_loss‚0.22820511847786984 1loss/FBt3/assignment_loss‚0.09205709397792816 1loss/SRqqt4/detection_loss‚1.8846999794244765 0loss/SRqqt2/assignment_loss‚1.42061110072442 1loss/FBt4/assignment_loss‚0.48810617439448833 -loss/FRt4/detection_loss‚2.32360139158037 trainer/global_step‚2449 0loss/FBt1/assignment_loss‚0.6831441876979975 .loss/FRt2/detection_loss‚1.709027082179532 1loss/SRqqt1/detection_loss‚2.3192520663142204 /loss/FBt1/detection_loss‚1.8290093334821553 0loss/FRt3/assignment_loss‚3.7321796110698156 1loss/SRqqt3/assignment_loss‚2.080229789018631 /loss/FRt3/detection_loss‚2.7311186705316817 epoch‚2 .loss/FRt1/assignment_loss‚3.95056055170117 /loss/FRt2/assignment_loss‚2.284460037946701 /loss/SRqqt2/detection_loss‚1.11100714442295 0loss/SRqqt3/detection_loss‚2.074564549733292 /loss/FBt4/detection_loss‚2.6508993932179044 %loss/total_loss‚48.23177013429718 _runtime‚2377.254091746 2loss/SRqqt1/assignment_loss‚2.2010303838178515 /loss/FRt1/detection_loss‚2.7875570663900087 .loss/FBt3/detection_loss‚3.185302197933197 _step‚201 /loss/FRt4/assignment_loss‚3.933034423987071 2loss/SRqqt4/assignment_loss‚1.8281068921089172”%ÑIYŽÓ /loss/FRt4/assignment_loss‚3.933034423987071 2loss/SRqqt4/assignment_loss‚1.8281068921089172 /loss/FBt2/detection_loss‚0.7040058941645331 % _timestamp‚1.7884189065042598e+09 1loss/FBt2/assignment_loss‚0.22820511847786984 1loss/FBt3/assignment_loss‚0.09205709397792816 1loss/SRqqt4/detection_loss‚1.8846999794244765 0loss/SRqqt2/assignment_loss‚1.42061110072442 1loss/FBt4/assignment_loss‚0.48810617439448833 -loss/FRt4/detection_loss‚2.32360139158037 trainer/global_step‚2449 0loss/FBt1/assignment_loss‚0.6831441876979975 .loss/FRt2/detection_loss‚1.709027082179532 1loss/SRqqt1/detection_loss‚2.3192520663142204 /loss/FBt1/detection_loss‚1.8290093334821553 0loss/FRt3/assignment_loss‚3.7321796110698156 1loss/SRqqt3/assignment_loss‚2.080229789018631 /loss/FRt3/detection_loss‚2.7311186705316817 epoch‚2 .loss/FRt1/assignment_loss‚3.95056055170117 /loss/FRt2/assignment_loss‚2.284460037946701 /loss/SRqqt2/detection_loss‚1.11100714442295 0loss/SRqqt3/detection_loss‚2.074564549733292 /loss/FBt4/detection_loss‚2.6508993932179044 %loss/total_loss‚48.23177013429718 _runtime‚2377.254091746 2loss/SRqqt1/assignment_loss‚2.2010303838178515 /loss/FRt1/detection_loss‚2.7875570663900087 .loss/FBt3/detection_loss‚3.185302197933197 _step‚201Éñ Ñš‚(: â¶äÔù ¡x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.46' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.491999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿk Êe‚(:Ú â¶äÔä‚Îz disk.dm-0.in‚ 4.9296875 disk.sda2.in‚0 disk.sda.out‚180.875# proc.memory.rssMB‚ 2863.27734375 disk.rbd1.in‚0 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.dm-0.out‚ 10139.8828125' proc.memory.availableMB‚ 341.3671875 disk.sda3.in‚7.1875 network.sent‚ 4.935271e+06 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 111.763671875 disk.nvme0c0n1.in‚ 4.9296875$ disk.nvme0c0n1.out‚ 10139.8828125 disk.sda1.in‚ 0.05078125" disk.nvme0n1.out‚ 10139.8828125 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda.in‚ 7.23828125) proc.memory.percent‚17.47605800628662 cpu‚25.297567222533484 network.recv‚858395 disk.dm-1.in‚7.1875 disk.sda3.out‚ 180.8359375 disk.nvme0n1.in‚ 4.9296875$ memory_percent‚97.91646003723145 proc.cpu.threads‚17) disk./.usagePercent‚73.66809341030873 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 180.8359375 disk.sda1.out‚0% disk./.usageGB‚1097.7078285217285 disk.rbd1.out‚ 1.16357421875ÖÞ»lš‘‚(: ñ¶äÔÉö’x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.586' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.717200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v2t’‚(:é ñ¶äÔÕÊ÷z disk.rbd3.in‚0 disk.sda3.in‚7.1875 disk.rbd2.in‚0 network.sent‚ 5.043899e+06 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.sda.in‚ 7.23828125 disk.rbd1.in‚0& proc.memory.availableMB‚ 341.359375 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.nvme0n1.in‚ 4.9296875% disk.nvme0n1.out‚10140.5478515625) proc.memory.percent‚17.47605800628662 disk.rbd2.out‚ 111.763671875 disk.rbd0.out‚0 disk.dm-1.in‚7.1875 disk.sda.out‚ 181.43359375 disk.sda3.out‚ 181.39453125# proc.memory.rssMB‚ 2863.27734375 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.out‚0 disk.rbd0.in‚0 network.recv‚864793 disk.sda2.in‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 cpu‚25.295926790659717' disk.nvme0c0n1.out‚10140.5478515625$ memory_percent‚97.91650772094727) disk./.usagePercent‚73.70179757021405% disk./.usageGB‚1098.2100448608398 disk.drbd1777.in‚ 1.03515625 disk.rbd1.out‚ 1.16357421875" disk.dm-0.out‚10140.5478515625 disk.dm-1.out‚ 181.39453125 disk.drbd1777.out‚0_/›“‚(: €·äÔªŽ®x gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.356' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.871199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g¥Å“o”‚(:ä €·äÔÈÇöz" disk.dm-0.out‚10239.6669921875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.in‚ 7.23828125 disk.nvme0n1.in‚ 4.9296875% disk.nvme0n1.out‚10239.6669921875 network.recv‚868153 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.1875) proc.memory.percent‚17.47605800628662 disk.drbd1777.in‚ 1.03515625' disk.nvme0c0n1.out‚10239.6669921875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2265625 cpu‚25.294416148873303 disk.dm-1.in‚7.1875 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.dm-1.out‚ 182.01171875 disk.rbd2.out‚ 111.888671875) disk./.usagePercent‚73.70168006275584 disk.dm-0.in‚ 4.9296875 disk.sda2.in‚0 disk.rbd4.out‚0 disk.sda.out‚ 182.05078125$ memory_percent‚97.91655540466309 network.sent‚ 5.04921e+06$ disk./.usageGB‚1098.208293914795 disk.sda3.out‚ 182.01171875# proc.memory.rssMB‚ 2863.27734375 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0' proc.memory.availableMB‚ 341.3515625zC2•- _wandbruntime‚2421  _runtime‚2421{ЙÆ–À trainer/global_step‚2499 $ _timestamp‚1.788418951012285e+09 _runtime‚2421.762339981 _step‚202 & lr-AdamW/pg1‚0.0007748142717921246 & lr-AdamW/pg2‚0.0007748142717921246]_DË—Å trainer/global_step‚2499 $ _timestamp‚1.788418951012285e+09 _runtime‚2421.762339981 _step‚202 & lr-AdamW/pg1‚0.0007748142717921246 & lr-AdamW/pg2‚0.0007748142717921246Êø·Qó2˜- _wandbruntime‚2421  _runtime‚2421o Ù²Ó'™ÍO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚103799022938112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27028236 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4728530147328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 _step‚203 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚99155664726016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43919107 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111296869 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚103799022938112 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46191589 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚111121552549888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65079609 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚103799269144064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46191591 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46191591 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4728801066496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111296869 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚103799022938112 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46191589 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚111121552549888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65079609 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚103799269144064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46191591 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46191591 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4728801066496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65130855 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27038591 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46209310 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46209308 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4645213909504 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43935369 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111164587310592 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16896780 _step‚204 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106434216038912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106434192098816 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43936002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27038589 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16897411 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚93924539183104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4631951887313 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4631728237961 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 _runtime‚2422.633109504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111314468 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚99194002670080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚103839444795904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4645213909504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4730668214784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111340165 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4730395211776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚98556250381705 $ _timestamp‚1.788418951882314e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111340165 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚103839444795904 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚93924522143744 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚99193984667136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111164884253696 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561ЕØá&ÛM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111340165 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4730395211776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚98556250381705 $ _timestamp‚1.788418951882314e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111340165 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚103839444795904 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚93924522143744 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚99193984667136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111164884253696 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4645442125824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65130855 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27038591 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46209310 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46209308 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4645213909504 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43935369 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111164587310592 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16896780 _step‚204 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106434216038912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106434192098816 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43936002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27038589 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16897411 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚93924539183104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4631951887313 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4631728237961 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 _runtime‚2422.633109504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111314468 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚99194002670080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚103839444795904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4645213909504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4730668214784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0Ì`EÃP2ž- _wandbruntime‚2422  _runtime‚2422ÈKÏ¿XŸÒ 2loss/SRqqt3/assignment_loss‚1.8659327959811145 0loss/FBt1/assignment_loss‚0.7409407351099487 1loss/SRqqt1/detection_loss‚2.0264181062579154 1loss/SRqqt2/detection_loss‚0.9572719920645741 1loss/SRqqt3/detection_loss‚1.9599508914080532 /loss/FBt2/detection_loss‚0.6572457406798313 $ _timestamp‚1.788418951884371e+09 0loss/FRt2/assignment_loss‚2.4694752483456224 1loss/SRqqt1/assignment_loss‚2.112824625521898 2loss/SRqqt2/assignment_loss‚1.2806862658985398 0loss/FBt2/assignment_loss‚0.2855570314739371 _runtime‚ 2422.63417972 0loss/FRt1/assignment_loss‚3.9689571508547155 .loss/FRt3/detection_loss‚2.589901572618729 epoch‚2 /loss/FRt1/detection_loss‚2.3281064502349715 1loss/SRqqt4/detection_loss‚2.0470650487475925 /loss/FBt3/detection_loss‚3.5618860870599747 1loss/SRqqt4/assignment_loss‚2.151957246661186 1loss/FBt3/assignment_loss‚0.30162906320765615 0loss/FBt4/assignment_loss‚0.3416520481308301 trainer/global_step‚2499 0loss/FRt4/assignment_loss‚3.9709926784038543 .loss/FRt4/detection_loss‚2.436653033643961 .loss/FBt1/detection_loss‚2.198306313582829 %loss/total_loss‚50.24474449603147 _step‚205 0loss/FRt3/assignment_loss‚4.6175062625836105 .loss/FRt2/detection_loss‚1.964193716093346 /loss/FBt4/detection_loss‚3.4096343914667764¶0n›] × 2loss/SRqqt3/assignment_loss‚1.8659327959811145 0loss/FBt1/assignment_loss‚0.7409407351099487 1loss/SRqqt1/detection_loss‚2.0264181062579154 1loss/SRqqt2/detection_loss‚0.9572719920645741 1loss/SRqqt3/detection_loss‚1.9599508914080532 /loss/FBt2/detection_loss‚0.6572457406798313 $ _timestamp‚1.788418951884371e+09 0loss/FRt2/assignment_loss‚2.4694752483456224 1loss/SRqqt1/assignment_loss‚2.112824625521898 2loss/SRqqt2/assignment_loss‚1.2806862658985398 0loss/FBt2/assignment_loss‚0.2855570314739371 _runtime‚ 2422.63417972 0loss/FRt1/assignment_loss‚3.9689571508547155 .loss/FRt3/detection_loss‚2.589901572618729 epoch‚2 /loss/FRt1/detection_loss‚2.3281064502349715 1loss/SRqqt4/detection_loss‚2.0470650487475925 /loss/FBt3/detection_loss‚3.5618860870599747 1loss/SRqqt4/assignment_loss‚2.151957246661186 1loss/FBt3/assignment_loss‚0.30162906320765615 0loss/FBt4/assignment_loss‚0.3416520481308301 trainer/global_step‚2499 0loss/FRt4/assignment_loss‚3.9709926784038543 .loss/FRt4/detection_loss‚2.436653033643961 .loss/FBt1/detection_loss‚2.198306313582829 %loss/total_loss‚50.24474449603147 _step‚205 0loss/FRt3/assignment_loss‚4.6175062625836105 .loss/FRt2/detection_loss‚1.964193716093346 /loss/FBt4/detection_loss‚3.4096343914667764͵q ½š¡‚(: ·äÔøçºx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚98.991' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.798199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›.l¢‚(:á ·äÔëŽ÷z disk.rbd1.out‚ 1.2265625 disk.sda2.out‚ 0.0390625 disk.rbd3.out‚0 disk.sda.out‚ 182.33984375 disk.sda2.in‚0 disk.dm-1.in‚7.1875 disk.drbd1777.in‚ 1.03515625 disk.rbd3.in‚0 disk.sda.in‚ 7.23828125 disk.sda3.out‚ 182.30078125) proc.memory.percent‚17.47605800628662) disk./.usagePercent‚73.70167878271818 disk.rbd2.in‚0 disk.rbd4.in‚0% disk./.usageGB‚1098.2082748413086 disk.rbd1.in‚0! disk.dm-0.out‚10338.849609375 disk.rbd2.out‚ 113.416015625 disk.dm-0.in‚ 4.9296875 disk.dm-1.out‚ 182.30078125& disk.nvme0c0n1.out‚10338.849609375 network.recv‚872631 disk.nvme0n1.in‚ 4.9296875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda3.in‚7.1875 cpu‚25.293025072618125 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9296875# memory_percent‚97.9151725769043& proc.memory.availableMB‚ 341.578125# proc.memory.rssMB‚ 2863.27734375 proc.cpu.threads‚17$ disk.nvme0n1.out‚10338.849609375 disk.drbd1777.out‚0 network.sent‚ 5.055239e+06ÏÅz‹Œ£‚(: ž·äÔȵx gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.8' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.76 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡šo¤‚(:ä ž·äÔºæÁz disk.sda3.in‚7.1875 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 182.890625 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.28955078125 disk.nvme0n1.in‚ 4.9296875% disk.nvme0n1.out‚10438.1845703125 disk.sda2.out‚ 0.0390625 disk.nvme0c0n1.in‚ 4.9296875$ memory_percent‚97.91676998138428 proc.cpu.threads‚17 network.sent‚ 5.163581e+06 disk.sda3.out‚ 182.8515625 disk.drbd1777.out‚0) disk./.usagePercent‚73.70168031876338% disk./.usageGB‚1098.2082977294922' disk.nvme0c0n1.out‚10438.1845703125 cpu‚25.291457970226176 disk.dm-1.in‚7.1875 disk.rbd2.out‚ 113.478515625 disk.sda1.out‚0 disk.rbd3.in‚0) proc.memory.percent‚17.47608184814453 network.recv‚878827 disk.dm-0.in‚ 4.9296875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 proc.memory.rssMB‚ 2863.28125 disk.sda.in‚ 7.23828125" disk.dm-0.out‚10438.1845703125 disk.dm-1.out‚ 182.8515625 disk.rbd3.out‚0( proc.memory.availableMB‚ 341.31640625܆›¥‚(: ­·äÔ«†›x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.429' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.085800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õêPn¦‚(:ã ­·äÔÔ–ˆ| disk.sda2.in‚0 disk.rbd4.in‚0( disk.nvme0c0n1.out‚10438.99951171875 disk.rbd1.out‚ 1.28955078125% proc.memory.availableMB‚ 174.21875 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.out‚0$ memory_percent‚98.93665313720703 disk.dm-1.out‚ 183.328125 disk.drbd1777.out‚0 proc.memory.rssMB‚ 2863.28125 disk.sda1.in‚ 0.05078125) proc.memory.percent‚17.47608184814453 disk.sda.out‚ 183.3671875# disk.dm-0.out‚10438.99951171875 disk.dm-1.in‚7.1875 disk.nvme0c0n1.in‚ 4.9296875 network.recv‚882911 disk.rbd2.out‚ 113.494140625 disk.rbd4.out‚0 disk.sda3.in‚7.1875 disk.rbd1.in‚0 network.sent‚ 5.169006e+06 disk.nvme0n1.in‚ 4.9296875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.in‚ 7.23828125& disk.nvme0n1.out‚10438.99951171875 disk.rbd3.in‚0 disk.sda3.out‚ 183.328125 cpu‚25.289968993763193 proc.cpu.threads‚17) disk./.usagePercent‚73.70167289454488% disk./.usageGB‚1098.2081871032715 disk.dm-0.in‚ 4.9296875„€¸!2§- _wandbruntime‚2467  _runtime‚2467(ÚôǨÁ & lr-AdamW/pg1‚0.0007726679911186703 & lr-AdamW/pg2‚0.0007726679911186703 trainer/global_step‚2549 % _timestamp‚1.7884189965387497e+09 _runtime‚2467.288696231 _step‚206ü ÿüÌ©Æ & lr-AdamW/pg1‚0.0007726679911186703 & lr-AdamW/pg2‚0.0007726679911186703 trainer/global_step‚2549 % _timestamp‚1.7884189965387497e+09 _runtime‚2467.288696231 _step‚206Îtô_2ª- _wandbruntime‚2467  _runtime‚2467ì%Ù'«ÓO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105808065484288 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 % _timestamp‚1.7884189965395162e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105807819278336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17215498 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100425112676761 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66386180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113463721 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚101072543926272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27545986 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47077539 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44760802 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4821966236160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95703342578688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105807819278336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17214818 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108451507215360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4735503555072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66360511 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 283502080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105808065484288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4735503555072 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100424871988257 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66360511 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚113273473451520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _runtime‚2467.289644276 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4735275352064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307442176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113438050 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44761484 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚113273780893696 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47077539 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 trainer/global_step‚2549 _step‚207 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚101072543926272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 trainer/global_step‚2549 _step‚207 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚101072543926272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105807819278336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17215498 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100425112676761 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66386180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113463721 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚101072543926272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27545986 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47077539 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44760802 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4821966236160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95703342578688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105807819278336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17214818 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108451507215360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262ÏMؾ©2­- _wandbruntime‚2468  _runtime‚2468Ö=ÒbU®ÝM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47095258 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113507017 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275100160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 trainer/global_step‚2549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95739645036544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47095260 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101110899873280 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚101110899873280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105848241136128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27556339 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100463247534425 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66386062 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4737113046528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4823831778304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 _runtime‚2468.164075269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4737113046528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4737341262848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101110881870336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108492700851712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105847994916864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44777362 D>De%>AŽ viceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66411757 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4824106878464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44778014 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299040256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113481320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95739662075904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17221023 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113316807730176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100463006845713 % _timestamp‚1.7884189974134111e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66386062 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262ÁòDDè&¯âM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101110899873280 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚101110899873280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105848241136128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27556339 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100463247534425 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66386062 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4737113046528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4823831778304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 _runtime‚2468.164075269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4737113046528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4737341262848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101110881870336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108492700851712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105847994916864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44777362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66411757 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4824106878464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44778014 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299040256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113481320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95739662075904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17221023 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113316807730176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100463006845713 % _timestamp‚1.7884189974134111e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66386062 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47095258 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113507017 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275100160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 trainer/global_step‚2549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95739645036544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47095260 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0вðMO2°- _wandbruntime‚2468  _runtime‚2468µ0ÞX±Ò 0loss/FRt2/assignment_loss‚2.2862510960929248 2loss/SRqqt2/assignment_loss‚1.4001691957859144 /loss/FRt4/detection_loss‚2.2951247241045976 /loss/FRt3/assignment_loss‚4.071953150629997 0loss/FBt3/assignment_loss‚0.7825682796537876 1loss/SRqqt2/detection_loss‚1.1417655386540988 1loss/SRqqt4/detection_loss‚1.9625488804734272 %loss/total_loss‚52.17200669995068 epoch‚2 2loss/SRqqt3/assignment_loss‚1.9947539189336252 1loss/SRqqt4/assignment_loss‚2.674545522617257 .loss/FRt1/detection_loss‚2.331653011031449 2loss/SRqqt1/assignment_loss‚1.9750101119279861 /loss/FRt3/detection_loss‚2.6283148720860483 .loss/FBt1/detection_loss‚2.662669189274311 /loss/FRt2/detection_loss‚1.8497587922884493 /loss/FRt1/assignment_loss‚4.597105599939823 0loss/FBt4/assignment_loss‚0.4230971497793992 /loss/FBt3/detection_loss‚3.0715499073266983 _step‚209 /loss/FRt4/assignment_loss‚4.643011223625493 0loss/FBt1/assignment_loss‚1.1182440519332886 /loss/FBt4/detection_loss‚3.0140431026617684 _runtime‚2468.165142149 0loss/FBt2/assignment_loss‚0.2789956296860762 1loss/SRqqt1/detection_loss‚2.2178477258295626 0loss/SRqqt3/detection_loss‚1.976567700809362 % _timestamp‚1.7884189974153247e+09 /loss/FBt2/detection_loss‚0.7744583248053319 trainer/global_step‚2549æ[dÖ]²× /loss/FBt2/detection_loss‚0.7744583248053319 trainer/global_step‚2549 0loss/FRt2/assignment_loss‚2.2862510960929248 2loss/SRqqt2/assignment_loss‚1.4001691957859144 /loss/FRt4/detection_loss‚2.2951247241045976 /loss/FRt3/assignment_loss‚4.071953150629997 0loss/FBt3/assignment_loss‚0.7825682796537876 1loss/SRqqt2/detection_loss‚1.1417655386540988 1loss/SRqqt4/detection_loss‚1.9625488804734272 %loss/total_loss‚52.17200669995068 epoch‚2 2loss/SRqqt3/assignment_loss‚1.9947539189336252 1loss/SRqqt4/assignment_loss‚2.674545522617257 .loss/FRt1/detection_loss‚2.331653011031449 2loss/SRqqt1/assignment_loss‚1.9750101119279861 /loss/FRt3/detection_loss‚2.6283148720860483 .loss/FBt1/detection_loss‚2.662669189274311 /loss/FRt2/detection_loss‚1.8497587922884493 /loss/FRt1/assignment_loss‚4.597105599939823 0loss/FBt4/assignment_loss‚0.4230971497793992 /loss/FBt3/detection_loss‚3.0715499073266983 _step‚209 /loss/FRt4/assignment_loss‚4.643011223625493 0loss/FBt1/assignment_loss‚1.1182440519332886 /loss/FBt4/detection_loss‚3.0140431026617684 _runtime‚2468.165142149 0loss/FBt2/assignment_loss‚0.2789956296860762 1loss/SRqqt1/detection_loss‚2.2178477258295626 0loss/SRqqt3/detection_loss‚1.976567700809362 % _timestamp‚1.7884189974153247e+09Ñú®ÄÁ³‚(:… ¼·äÔ耭x gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.034' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚœPÇm´‚(:â ¼·äÔˆÑz disk.rbd3.in‚0) proc.memory.percent‚17.47608184814453 disk.rbd1.out‚ 1.28955078125 disk.sda2.in‚0 disk.sda3.out‚ 183.8828125 disk.rbd1.in‚0$ disk.nvme0n1.out‚10538.330078125! disk.rbd2.out‚113.64306640625 disk.rbd0.in‚0 network.sent‚ 5.174271e+06 disk.nvme0n1.in‚ 4.9296875 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0) disk./.usagePercent‚73.70174124855653 disk.sda.in‚ 7.23828125 disk.sda.out‚ 183.921875 disk.sda3.in‚7.1875 disk.dm-0.in‚ 4.9296875! disk.dm-0.out‚10538.330078125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.1875 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9296875& disk.nvme0c0n1.out‚10538.330078125 disk.sda1.out‚0' proc.memory.availableMB‚ 173.6953125 proc.memory.rssMB‚ 2863.28125 proc.cpu.threads‚17 disk.rbd3.out‚0$ memory_percent‚98.93984794616699 disk.rbd0.out‚0 disk.drbd1777.out‚0 cpu‚25.288559560502698 network.recv‚886634% disk./.usageGB‚1098.2092056274414 disk.dm-1.out‚ 183.8828125 disk.sda1.in‚ 0.05078125AD±Ö›µ‚(: Ë·äÔöï•x gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.974' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.594800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{úÊ p¶‚(:å Ë·äÔ±ýôz disk.rbd3.out‚0 proc.cpu.threads‚17 network.sent‚ 5.28267e+06 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd1.out‚ 1.28955078125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 184.4921875 disk.sda3.out‚ 184.453125& proc.memory.availableMB‚ 173.453125) disk./.usagePercent‚73.70174124855653 disk.dm-1.in‚7.1875( disk.nvme0c0n1.out‚10637.69970703125 disk.sda2.out‚ 0.0390625 cpu‚25.287131678593273 disk.rbd3.in‚0 disk.nvme0n1.in‚ 4.9296875# disk.dm-0.out‚10637.69970703125$ memory_percent‚98.94132614135742 proc.memory.rssMB‚ 2863.28125 disk.dm-1.out‚ 184.453125 disk.rbd0.in‚0 disk.sda.in‚ 7.23828125 disk.rbd1.in‚0% disk./.usageGB‚1098.2092056274414! disk.rbd2.out‚113.65869140625 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda3.in‚7.1875& disk.nvme0n1.out‚10637.69970703125 disk.dm-0.in‚ 4.9296875) proc.memory.percent‚17.47608184814453 network.recv‚892800 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0{ÄÆ·‚(:… Ú·äÔ¶´‰x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XÔ3äm¸‚(:â Ú·äÔª†Ñz$ disk./.usageGB‚1098.208293914795# disk.dm-0.out‚10638.01611328125 disk.rbd0.out‚0 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0 disk.sda.in‚ 7.23828125 disk.sda.out‚ 184.96875 disk.nvme0n1.in‚ 4.9296875& disk.nvme0n1.out‚10638.01611328125 disk.sda1.out‚0 network.sent‚ 5.288526e+06 disk.dm-0.in‚ 4.9296875! disk.rbd2.out‚113.72119140625 disk.rbd1.out‚ 1.28955078125 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 proc.memory.rssMB‚ 2863.28125 disk.sda2.in‚0 disk.dm-1.in‚7.1875$ memory_percent‚98.94180297851562 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0# proc.memory.availableMB‚173.375) proc.memory.percent‚17.47608184814453 disk.sda3.in‚7.1875 disk.sda3.out‚ 184.9296875 disk.rbd0.in‚0 network.recv‚896904 disk.dm-1.out‚ 184.9296875 disk.rbd4.out‚0( disk.nvme0c0n1.out‚10638.01611328125 cpu‚25.285633674659064) disk./.usagePercent‚73.70168006275584÷å~æ2¹- _wandbruntime‚2512  _runtime‚2512Ì/\ź¿ & lr-AdamW/pg1‚0.0007704442885982214 & lr-AdamW/pg2‚0.0007704442885982214 trainer/global_step‚2599 % _timestamp‚1.7884190419569223e+09 _runtime‚ 2512.7069413 _step‚210Ä Ê»Ä _step‚210 & lr-AdamW/pg1‚0.0007704442885982214 & lr-AdamW/pg2‚0.0007704442885982214 trainer/global_step‚2599 % _timestamp‚1.7884190419569223e+09 _runtime‚ 2512.7069413Òk|Õ92¼- _wandbruntime‚2512  _runtime‚2512'*?ÆÙ'½ÓO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚102989459132416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67667082 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102989441129472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚211 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115630573 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17536134 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4915392898048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4915670108672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102331628452265 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102989441129472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚107816615618560 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115425384926208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚2512.707869709 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47963489 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚650 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2599 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚107816861824512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45599218 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17535484 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4813386629921 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚102989459132416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110510015968256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97518482510848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47963491 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45599870 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚652 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4813162980777 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47963489 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28063736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67641413 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67641413 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115425686076928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4827174489088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102331869140769 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4827402692096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4827402692096 % _timestamp‚1.7884190419576635e+09—}–±[¾ØO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47963491 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67667082 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28063734 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚107816861824512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110509992028160 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97518465471488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115604902 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ÓBgåƒGADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47963489 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚650 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2599 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚107816861824512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45599218 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17535484 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4813386629921 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚102989459132416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110510015968256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97518482510848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47963491 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45599870 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚652 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4813162980777 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47963489 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28063736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67641413 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67641413 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115425686076928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4827174489088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102331869140769 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4827402692096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4827402692096 % _timestamp‚1.7884190419576635e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚102989459132416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67667082 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102989441129472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚211 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115630573 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17536134 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4915392898048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4915670108672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102331628452265 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102989441129472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚107816615618560 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115425384926208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚2512.707869709 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47981210 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45617267 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67666964 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45616590 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚107857037476352 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115648172 trainer/global_step‚2599 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4829240399872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103027797076480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4829240399872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28074089 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110551185664512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67666964 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115673869 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4829012183552 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97554784968704 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚103027779073536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17543176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17542501 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296943104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67692659 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4829012183552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115648172 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97554767929344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28074091 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47981210 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45617267 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67666964 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45616590 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚107857037476352 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115648172 trainer/global_step‚2599 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4829240399872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103027797076480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4829240399872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28074089 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110551185664512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67666964 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115673869 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4829012183552 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97554784968704 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚103027779073536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17543176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17542501 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296943104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67692659 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4829012183552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115648172 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97554767929344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28074091 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824Ô¼í½b2Â- _wandbruntime‚2513  _runtime‚2513/òtmZÃÔ %loss/total_loss‚47.78423288419732 _step‚213 0loss/FRt3/assignment_loss‚3.3092249986973217 2loss/SRqqt4/assignment_loss‚1.9312799339200937 .loss/FRt2/detection_loss‚1.848289155849704 1loss/SRqqt3/detection_loss‚1.9251672217379445 /loss/FBt1/detection_loss‚1.9929701859752338 0loss/FBt1/assignment_loss‚0.5560969536293973 /loss/FRt3/detection_loss‚2.4313408859232637 .loss/FRt1/detection_loss‚2.475224607579765 /loss/FRt4/detection_loss‚2.9870017021894455 /loss/FBt2/detection_loss‚0.6733967459431904 .loss/FBt3/detection_loss‚3.714394688606262 epoch‚2 0loss/FRt1/assignment_loss‚3.6649168786548434 0loss/FRt2/assignment_loss‚2.1484328629793943 /loss/FBt4/detection_loss‚2.1585926158087596 2loss/SRqqt2/assignment_loss‚1.3702004349480073 trainer/global_step‚2599 % _timestamp‚1.7884190428228307e+09 /loss/FRt4/assignment_loss‚4.280913919210434 0loss/FBt4/assignment_loss‚0.4077935814857483 1loss/SRqqt1/detection_loss‚2.3761941269040108 0loss/SRqqt4/detection_loss‚1.987335503685708 _runtime‚2513.572664621 1loss/SRqqt2/detection_loss‚0.9365058501561483 1loss/SRqqt1/assignment_loss‚2.077845169696957 1loss/SRqqt3/assignment_loss‚2.052986479323843 1loss/FBt2/assignment_loss‚0.21879355631254238 1loss/FBt3/assignment_loss‚0.25933482497930527ŸuL×_ÄÙ .loss/FRt1/detection_loss‚2.475224607579765 /loss/FRt4/detection_loss‚2.9870017021894455 /loss/FBt2/detection_loss‚0.6733967459431904 .loss/FBt3/detection_loss‚3.714394688606262 epoch‚2 0loss/FRt1/assignment_loss‚3.6649168786548434 0loss/FRt2/assignment_loss‚2.1484328629793943 /loss/FBt4/detection_loss‚2.1585926158087596 2loss/SRqqt2/assignment_loss‚1.3702004349480073 trainer/global_step‚2599 % _timestamp‚1.7884190428228307e+09 /loss/FRt4/assignment_loss‚4.280913919210434 0loss/FBt4/assignment_loss‚0.4077935814857483 1loss/SRqqt1/detection_loss‚2.3761941269040108 0loss/SRqqt4/detection_loss‚1.987335503685708 _runtime‚2513.572664621 1loss/SRqqt2/detection_loss‚0.9365058501561483 1loss/SRqqt1/assignment_loss‚2.077845169696957 1loss/SRqqt3/assignment_loss‚2.052986479323843 1loss/FBt2/assignment_loss‚0.21879355631254238 1loss/FBt3/assignment_loss‚0.25933482497930527 %loss/total_loss‚47.78423288419732 _step‚213 0loss/FRt3/assignment_loss‚3.3092249986973217 2loss/SRqqt4/assignment_loss‚1.9312799339200937 .loss/FRt2/detection_loss‚1.848289155849704 1loss/SRqqt3/detection_loss‚1.9251672217379445 /loss/FBt1/detection_loss‚1.9929701859752338 0loss/FBt1/assignment_loss‚0.5560969536293973 /loss/FRt3/detection_loss‚2.4313408859232637Õ¯âãÅ‚(:„ é·äÔ¼Œ˜x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚97.112' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.4224 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›e¦nÆ‚(:ã é·äÔÔ­·z network.sent‚ 5.294826e+06" disk.dm-0.out‚10737.3173828125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.1875 disk.dm-1.out‚ 185.34375 disk.rbd0.in‚0 cpu‚25.28424987029112 disk.rbd4.in‚0( disk./.usagePercent‚73.7017456006846 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9296875 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd3.out‚0' disk.nvme0c0n1.out‚10737.3173828125 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda2.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚115.24853515625$ memory_percent‚98.94130229949951$ disk./.usageGB‚1098.209270477295 disk.sda3.out‚ 185.34375 disk.rbd1.out‚ 1.28955078125 disk.sda.in‚ 7.23828125 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.sda.out‚ 185.3828125( proc.memory.availableMB‚ 173.45703125% disk.nvme0n1.out‚10737.3173828125 disk.dm-0.in‚ 4.9296875 network.recv‚901470 disk.sda3.in‚7.1875 disk.rbd3.in‚0# proc.memory.rssMB‚ 2863.28515625) proc.memory.percent‚17.47610569000244”=‹|RÇ‚(: ø·äÔšó¢y gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.683' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.936600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593¯–¡H! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gô9vÈ‚(:ë ø·äÔøÚËz disk.rbd0.in‚0" disk.dm-0.out‚10836.3837890625 disk.drbd1777.out‚0) proc.memory.percent‚17.47610569000244 disk.rbd1.out‚ 1.28955078125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 proc.cpu.threads‚17 network.sent‚ 5.402572e+06 disk.sda1.out‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2863.28515625 disk.sda.out‚ 185.94140625$ disk./.usageGB‚1098.209270477295 disk.nvme0c0n1.in‚ 4.9296875 disk.sda.in‚ 7.23828125 disk.dm-1.in‚7.1875 disk.dm-1.out‚ 185.90234375! disk.rbd2.out‚115.37353515625 disk.rbd3.in‚0 disk.sda3.out‚ 185.90234375 disk.rbd1.in‚0% disk.nvme0n1.out‚10836.3837890625 disk.rbd4.out‚0$ memory_percent‚98.94077777862549 cpu‚25.282777246441054' disk.nvme0c0n1.out‚10836.3837890625 disk.sda3.in‚7.1875 disk.dm-0.in‚ 4.9296875 disk.sda2.in‚0 disk.rbd4.in‚0( proc.memory.availableMB‚ 173.54296875 network.recv‚906738( disk./.usagePercent‚73.7017456006846 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.nvme0n1.in‚ 4.9296875 disk.sda2.out‚ 0.0390625Âk¥ÍÉ‚(:„ ‡¸äÔȦ£x gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.623 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d‰$ðgÊ‚(:Ü ‡¸äÔçöÔ{ disk.nvme0n1.in‚ 4.9296875 disk.sda2.in‚0 disk.sda3.in‚7.1875 disk.dm-1.out‚ 186.359375! disk.rbd2.out‚115.38525390625 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 186.359375" proc.memory.availableMB‚173.25 disk.rbd1.in‚0# disk.nvme0n1.out‚10935.49609375 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.23828125 network.recv‚911287 disk.drbd1777.in‚ 1.03515625# proc.memory.rssMB‚ 2863.28515625) proc.memory.percent‚17.47610569000244 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 cpu‚25.281400391634055) disk./.usagePercent‚73.70174713672981% disk./.usageGB‚1098.2092933654785 disk.rbd1.out‚ 1.28955078125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.1875% disk.nvme0c0n1.out‚10935.49609375 disk.sda.out‚ 186.3984375$ memory_percent‚98.94256591796875 network.sent‚ 5.408726e+06 disk.dm-0.in‚ 4.9296875 disk.dm-0.out‚10935.49609375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.out‚0NAs2Ë- _wandbruntime‚2557  _runtime‚2557×}ÞôÇÌÁ & lr-AdamW/pg1‚0.0007681436276470503 & lr-AdamW/pg2‚0.0007681436276470503 trainer/global_step‚2649 % _timestamp‚1.7884190870203192e+09 _runtime‚2557.770244402 _step‚214ª^$ÌÍÆ % _timestamp‚1.7884190870203192e+09 _runtime‚2557.770244402 _step‚214 & lr-AdamW/pg1‚0.0007681436276470503 & lr-AdamW/pg2‚0.0007681436276470503 trainer/global_step‚2649ÖÒؼ¬2Î- _wandbruntime‚2557  _runtime‚2557©l(Ù'ÏÓO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5008912894464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275113472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2649 _runtime‚2557.771211083 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109825411958784 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚104238384916273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48849439 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68922315 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28581484 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚109825658164736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68947984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4919073626112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚660 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚104906338332672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48849439 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17858679 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚99333605403648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4905020201129 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109825658164736 % _timestamp‚1.7884190870210514e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4919301829120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17858019 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚104906356335616 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117797425 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚104906356335616 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48849441 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28581486 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚662 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17858019 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚104906356335616 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117797425 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚104906356335616 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48849441 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28581486 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚662 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109825411958784 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚104238384916273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48849439 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68922315 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28581484 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚109825658164736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68947984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4919073626112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚660 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚104906338332672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48849439 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17858679 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚99333605403648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262×úÅ®c2Ñ- _wandbruntime‚2558  _runtime‚2558ç÷ô2#ÒÝM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4906628951585 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 trainer/global_step‚2649 _step‚216 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68947866 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109865587597312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117815024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104276760462441 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4920911320576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28591839 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5010775897088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46456797 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117840721 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17864318 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301137408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5011053094400 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68973561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277197312 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4920911320576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48867158 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46456157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99369890822144 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109865833816576 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚99369907861504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17864956 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117620422434304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112609670477312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 % _timestamp‚1.7884190878832297e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109865833816576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4906852600937 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚104944676276736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117815024 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117840721 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104944694279680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4921139536896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104276519773729r*Øè&ÓâM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48867160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68947866 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48867160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104944676276736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112609646537216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48867158 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68973561 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4920911320576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48867158 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46456157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99369890822144 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109865833816576 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚99369907861504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17864956 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117620422434304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112609670477312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 % _timestamp‚1.7884190878832297e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109865833816576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4906852600937 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚104944676276736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117815024 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117840721 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104944694279680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4921139536896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104276519773729 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4906628951585 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 trainer/global_step‚2649 _step‚216 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68947866 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109865587597312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117815024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104276760462441 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4920911320576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28591839 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5010775897088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46456797 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117840721 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17864318 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301137408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5011053094400 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68973561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277197312 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 % _timestamp‚1.7884191319763262e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5010972763136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _runtime‚2602.726430147 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111834454504960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚106823253538816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70203217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111834208299008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4996653772337 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5011200966144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49735389 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47277990 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111834208299008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4996653772337 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5011200966144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49735389 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47277990 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 % _timestamp‚1.7884191319763262e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5010972763136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _runtime‚2602.726430147ÛÒݺ2ã- _wandbruntime‚2603  _runtime‚2603ƒ³èå&äßM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚106861591482880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120007573 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47294542 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚220 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101185030754304 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111874630156800 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70228768 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111874383937536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70228768 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70254463 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5013038673920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119772809151488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101185013714944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5012810457600 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70254463 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4998262522793 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49753108 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29109591 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119772514305536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5104653861376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚106861573479936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49753110 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5104382955520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49753108 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111874630156800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114668155290112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5013038673920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119772809151488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101185013714944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5012810457600 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70254463 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4998262522793 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49753108 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29109591 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119772514305536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5104653861376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚106861573479936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49753110 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5104382955520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49753108 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111874630156800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114668155290112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120007573 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47294542 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚220 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101185030754304 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111874630156800 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70228768 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111874383937536ÜND¿e2æ- _wandbruntime‚2603  _runtime‚2603ªœnìVçÐ /loss/SRqqt1/detection_loss‚2.16589792690626 epoch‚2 _runtime‚2603.589669024 /loss/FBt2/detection_loss‚0.5724530939811042 2loss/SRqqt1/assignment_loss‚2.2391102344888014 1loss/SRqqt4/assignment_loss‚2.385424028302348 1loss/FBt2/assignment_loss‚0.23262477751016927 /loss/FBt1/detection_loss‚2.5715919733047485 /loss/FRt2/detection_loss‚1.7322283707699686 /loss/FRt3/detection_loss‚2.5604920448401036 .loss/FRt4/detection_loss‚2.551686965756946 .loss/FBt4/detection_loss‚2.244318723678589 &loss/total_loss‚52.012464713564725 trainer/global_step‚2699 _step‚221 0loss/FBt4/assignment_loss‚0.4442181023122329 1loss/SRqqt3/assignment_loss‚1.731424475312233 /loss/SRqqt4/detection_loss‚2.27977789903796 1loss/FBt3/assignment_loss‚0.48514706480782477 /loss/FBt3/detection_loss‚4.3809823274612425 /loss/FRt1/detection_loss‚2.6496081590652465 0loss/FBt1/assignment_loss‚0.8419187764326731 1loss/SRqqt2/detection_loss‚1.1552797008019227 0loss/SRqqt3/detection_loss‚2.042552835345268 % _timestamp‚1.7884191328398662e+09 /loss/FRt1/assignment_loss‚4.143142986297607 0loss/FRt3/assignment_loss‚3.8962517044483085 1loss/SRqqt2/assignment_loss‚1.506012392273316 0loss/FRt2/assignment_loss‚2.3753766046380096 /loss/FRt4/assignment_loss‚4.824943545791838Œó[èÕ 1loss/SRqqt3/assignment_loss‚1.731424475312233 /loss/SRqqt4/detection_loss‚2.27977789903796 1loss/FBt3/assignment_loss‚0.48514706480782477 /loss/FBt3/detection_loss‚4.3809823274612425 /loss/FRt1/detection_loss‚2.6496081590652465 0loss/FBt1/assignment_loss‚0.8419187764326731 1loss/SRqqt2/detection_loss‚1.1552797008019227 0loss/SRqqt3/detection_loss‚2.042552835345268 % _timestamp‚1.7884191328398662e+09 /loss/FRt1/assignment_loss‚4.143142986297607 0loss/FRt3/assignment_loss‚3.8962517044483085 1loss/SRqqt2/assignment_loss‚1.506012392273316 0loss/FRt2/assignment_loss‚2.3753766046380096 /loss/FRt4/assignment_loss‚4.824943545791838 /loss/SRqqt1/detection_loss‚2.16589792690626 epoch‚2 _runtime‚2603.589669024 /loss/FBt2/detection_loss‚0.5724530939811042 2loss/SRqqt1/assignment_loss‚2.2391102344888014 1loss/SRqqt4/assignment_loss‚2.385424028302348 1loss/FBt2/assignment_loss‚0.23262477751016927 /loss/FBt1/detection_loss‚2.5715919733047485 /loss/FRt2/detection_loss‚1.7322283707699686 /loss/FRt3/detection_loss‚2.5604920448401036 .loss/FRt4/detection_loss‚2.551686965756946 .loss/FBt4/detection_loss‚2.244318723678589 &loss/total_loss‚52.012464713564725 trainer/global_step‚2699 _step‚221 0loss/FBt4/assignment_loss‚0.4442181023122329Ý"•‚é‚(:… øäÔ‡ú«z gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚122.149' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Túonê‚(:ã øäÔéšÓ| disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.9296875 network.sent‚ 5.534735e+06 disk.drbd1777.in‚ 1.03515625 disk.sda.in‚ 7.23828125 disk.sda3.in‚7.1875( proc.memory.availableMB‚ 173.34765625) disk./.usagePercent‚73.70171334373529 disk.sda2.in‚0 disk.dm-1.in‚7.1875 disk.rbd2.out‚117.1474609375 disk.sda3.out‚ 188.8828125 disk.rbd1.out‚ 1.28955078125$ disk.nvme0n1.out‚11135.447265625 disk.drbd1777.out‚0" memory_percent‚98.941969871521# proc.memory.rssMB‚ 2863.28515625 disk.sda.out‚ 188.921875 disk.rbd0.out‚0% disk./.usageGB‚1098.2087898254395 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚0) proc.memory.percent‚17.47610569000244 proc.cpu.threads‚17! disk.dm-0.out‚11135.447265625 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd3.out‚0 cpu‚25.276114331977993 network.recv‚930183 disk.dm-1.out‚ 188.8828125 disk.rbd4.in‚0 disk.rbd4.out‚0& disk.nvme0c0n1.out‚11135.447265625 l6*‹ë‚(:€ Ò¸äÔЂ x gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.5' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+fÞþqì‚(:æ Ò¸äÔتÉz disk.dm-1.out‚ 189.296875 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 1.28955078125 cpu‚25.274876303188453 disk.dm-0.in‚ 4.9296875 disk.dm-1.in‚7.1875 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.nvme0n1.in‚ 4.9296875 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625& proc.memory.availableMB‚ 173.234375 disk.drbd1777.out‚0 disk.sda.out‚ 189.3359375# proc.memory.rssMB‚ 2863.28515625) proc.memory.percent‚17.47610569000244 proc.cpu.threads‚17 network.recv‚936176) disk./.usagePercent‚73.70171334373529' disk.nvme0c0n1.out‚11234.5830078125 disk.sda3.in‚7.1875$ memory_percent‚98.94266128540039 network.sent‚ 5.642972e+06% disk./.usageGB‚1098.2087898254395 disk.rbd2.in‚0! disk.rbd2.out‚117.30029296875% disk.nvme0n1.out‚11234.5830078125" disk.dm-0.out‚11234.5830078125 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.23828125 disk.sda3.out‚ 189.296875 disk.rbd1.in‚0¯Ÿy í‚(:„ á¸äÔƧ x gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.681' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.936¹d}2 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SËKsî‚(:è á¸äÔÞœÌ{# proc.memory.rssMB‚ 2863.28515625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 189.7421875 disk.rbd1.out‚ 1.28955078125 network.sent‚ 5.649362e+06) disk./.usagePercent‚73.70171334373529 disk.rbd4.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚11333.83935546875 disk.dm-0.in‚ 4.9296875& proc.memory.availableMB‚ 173.140625 disk.rbd4.in‚0( disk.nvme0c0n1.out‚11333.83935546875 disk.rbd3.out‚0 disk.sda.in‚ 7.23828125 disk.sda3.in‚7.1875 disk.rbd1.in‚0# disk.dm-0.out‚11333.83935546875 disk.sda2.out‚ 0.0390625 network.recv‚941057 disk.drbd1777.out‚0$ memory_percent‚98.94323348999023 proc.cpu.threads‚17 disk.sda.out‚ 189.78125 disk.sda2.in‚0! disk.rbd2.out‚117.31591796875) proc.memory.percent‚17.47610569000244 cpu‚25.27359128180173 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd3.in‚0 disk.dm-1.in‚7.1875 disk.rbd2.in‚0% disk./.usageGB‚1098.2087898254395 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚ 4.9296875 disk.dm-1.out‚ 189.7421875¼èt2ï- _wandbruntime‚2647  _runtime‚2647d”Åð¿ % lr-AdamW/pg1‚0.000763313364207821 % lr-AdamW/pg2‚0.000763313364207821 trainer/global_step‚2749 % _timestamp‚1.7884191769696286e+09 _runtime‚2647.719546512 _step‚222û­VŽÊñÄ % _timestamp‚1.7884191769696286e+09 _runtime‚2647.719546512 _step‚222 % lr-AdamW/pg1‚0.000763313364207821 % lr-AdamW/pg2‚0.000763313364207821 trainer/global_step‚2749ÞCÀæ’2ò- _wandbruntime‚2647  _runtime‚2647-`˜Û'óÕO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚644 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116685446466560 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50621339 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18500221 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48116561 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113843004639232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113843250845184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29616986 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18499577 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102963834149888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71509788 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5088287343545 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29616984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113843250845184 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5196300815360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚108740132739072 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 trainer/global_step‚2749 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5088063694401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102963851189248 % _timestamp‚1.7884191769703448e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5103100103168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268822016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5196031993344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292762112 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108051897844289 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5103100103168 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5196300815360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚108740132739072 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 trainer/global_step‚2749 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5088063694401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102963851189248 % _timestamp‚1.7884191769703448e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5103100103168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268822016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5196031993344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292762112 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108051897844289 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5103100103168 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113843004639232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113843250845184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29616986 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18499577 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102963834149888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71509788 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5088287343545 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29616984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113843250845184ßl¬d2õ- _wandbruntime‚2648  _runtime‚2648ßì7œå&ößM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48133132 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 % _timestamp‚1.7884191778297167e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50639060 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚108778488686080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5089896094001 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113883426497024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5104709594624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29627339 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 307428864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116726640102912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚108778470683136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122148728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122148728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚108778470683136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50639058 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122174425 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚103000136607744 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121924810477568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18506445 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5104937810944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116726616162816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71535365 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚108778488686080 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5104937810944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50639058 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121924503048704 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122174425 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71509670 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113883180277760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5197886885888 _step‚224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 _runtime‚2648.580448093 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121924503048704 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122174425 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71509670 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113883180277760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 ]KDeviceStatsMonitors´±³ì$.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5197886885888 _step‚224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 _runtime‚2648.580448093 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48133132 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 % _timestamp‚1.7884191778297167e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50639060 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚108778488686080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5089896094001 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113883426497024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5104709594624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29627339 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 307428864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116726640102912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚108778470683136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122148728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122148728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚108778470683136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50639058 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122174425 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚103000136607744 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121924810477568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18506445 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5104937810944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116726616162816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71535365 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚108778488686080 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5104937810944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50639058àŠ˜à2ø- _wandbruntime‚2648  _runtime‚26488ô°SùÍ 0loss/FBt1/assignment_loss‚0.5701781734824181 .loss/FRt1/detection_loss‚2.562970184499309 /loss/FBt1/detection_loss‚2.2204744424670935 %loss/total_loss‚50.13157615943015 0loss/FRt4/assignment_loss‚3.7102865207762945 /loss/FBt4/assignment_loss‚0.666309398852966 /loss/FRt4/detection_loss‚2.3609779462927865 .loss/FBt4/detection_loss‚2.602555350823836 2loss/SRqqt3/assignment_loss‚1.7394087342140467 1loss/SRqqt4/assignment_loss‚1.801052750647068 /loss/FRt1/assignment_loss‚4.320038954416911 0loss/FBt2/assignment_loss‚0.2088509096687459 /loss/FRt2/detection_loss‚1.7639618281354295 .loss/FRt3/detection_loss‚2.723373989801149 1loss/SRqqt4/detection_loss‚1.8108849227428436 _step‚225 trainer/global_step‚2749 $ _timestamp‚1.788419177831688e+09 /loss/FRt2/assignment_loss‚2.804516746642742 epoch‚2 1loss/SRqqt1/assignment_loss‚2.164972285181284 1loss/SRqqt2/assignment_loss‚1.718071440477734 1loss/SRqqt2/detection_loss‚1.1859304768883663 /loss/FBt2/detection_loss‚0.6522880833637849 _runtime‚2648.581503551 0loss/FRt3/assignment_loss‚3.3741695139859176 0loss/FBt3/assignment_loss‚0.8755337198575338 0loss/SRqqt1/detection_loss‚2.248039869964123 1loss/SRqqt3/detection_loss‚1.9317520001660222 .loss/FBt3/detection_loss‚4.114977916081746¨I ?XúÒ /loss/FRt4/detection_loss‚2.3609779462927865 .loss/FBt4/detection_loss‚2.602555350823836 2loss/SRqqt3/assignment_loss‚1.7394087342140467 1loss/SRqqt4/assignment_loss‚1.801052750647068 /loss/FRt1/assignment_loss‚4.320038954416911 0loss/FBt2/assignment_loss‚0.2088509096687459 /loss/FRt2/detection_loss‚1.7639618281354295 .loss/FRt3/detection_loss‚2.723373989801149 1loss/SRqqt4/detection_loss‚1.8108849227428436 _step‚225 trainer/global_step‚2749 $ _timestamp‚1.788419177831688e+09 /loss/FRt2/assignment_loss‚2.804516746642742 epoch‚2 1loss/SRqqt1/assignment_loss‚2.164972285181284 1loss/SRqqt2/assignment_loss‚1.718071440477734 1loss/SRqqt2/detection_loss‚1.1859304768883663 /loss/FBt2/detection_loss‚0.6522880833637849 _runtime‚2648.581503551 0loss/FRt3/assignment_loss‚3.3741695139859176 0loss/FBt3/assignment_loss‚0.8755337198575338 0loss/SRqqt1/detection_loss‚2.248039869964123 1loss/SRqqt3/detection_loss‚1.9317520001660222 .loss/FBt3/detection_loss‚4.114977916081746 0loss/FBt1/assignment_loss‚0.5701781734824181 .loss/FRt1/detection_loss‚2.562970184499309 /loss/FBt1/detection_loss‚2.2204744424670935 %loss/total_loss‚50.13157615943015 0loss/FRt4/assignment_loss‚3.7102865207762945 /loss/FBt4/assignment_loss‚0.666309398852966በóõû‚(:„ ð¸äÔõ°šx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.471' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kþ_zü‚(:ï ð¸äÔÏÓÎ{ disk.rbd3.out‚0 disk.drbd1777.in‚ 1.03515625 network.recv‚944829% disk./.usageGB‚1098.2087898254395 disk.sda.in‚ 7.23828125 disk.rbd4.in‚0$ memory_percent‚98.94223213195801# proc.memory.rssMB‚ 2863.28515625) proc.memory.percent‚17.47610569000244& disk.nvme0n1.out‚11433.00341796875 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 190.30859375! disk.rbd2.out‚117.31591796875( disk.nvme0c0n1.out‚11433.00341796875 disk.sda1.in‚ 0.05078125 disk.rbd1.in‚0 disk.dm-1.in‚7.1875 disk.rbd4.out‚0 network.sent‚ 5.654667e+06# disk.dm-0.out‚11433.00341796875' proc.memory.availableMB‚ 173.3046875 disk.sda1.out‚0 disk.sda3.in‚7.1875 disk.rbd1.out‚ 1.28955078125 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚73.70171334373529 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd3.in‚0 disk.sda3.out‚ 190.30859375 disk.nvme0n1.in‚ 4.9296875 disk.drbd1777.out‚0 disk.sda.out‚ 190.34765625 disk.sda2.in‚0 cpu‚25.272340170180936 proc.cpu.threads‚17¿ùûçý‚(:… ÿ¸äÔÅÐx gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 щXfþ‚(:Û ÿ¸äÔƱÈz proc.cpu.threads‚17 network.sent‚ 5.762987e+06 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.1875 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0$ memory_percent‚98.94223213195801 cpu‚25.271030928709667 network.recv‚950931 disk.dm-0.in‚ 4.9296875 disk.dm-0.out‚11433.32421875 disk.sda2.in‚0! disk.rbd2.out‚117.39404296875) proc.memory.percent‚17.47610569000244 disk.dm-1.out‚ 190.9609375 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.rbd3.in‚0% disk./.usageGB‚1098.2087898254395 disk.rbd4.out‚0% disk.nvme0c0n1.out‚11433.32421875 disk.sda.out‚191 disk.sda3.in‚7.1875 disk.rbd1.out‚ 1.28955078125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda3.out‚ 190.9609375 disk.rbd1.in‚0) disk./.usagePercent‚73.70171334373529 disk.nvme0n1.in‚ 4.9296875# disk.nvme0n1.out‚11433.32421875 disk.sda.in‚ 7.23828125' proc.memory.availableMB‚ 173.3046875# proc.memory.rssMB‚ 2863.28515625¿t |ÿ‚(:… Ž¹äÔÿ©‡x gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rÀ$j€‚(:ß Ž¹äÔǤÔz! disk.rbd2.out‚118.90576171875 disk.rbd4.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 1.28955078125( proc.memory.availableMB‚ 173.05078125 network.recv‚955087 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚ 4.9296875 disk.sda2.in‚0$ memory_percent‚98.94378185272217 cpu‚25.269806332304068 disk.nvme0c0n1.in‚ 4.9296875 disk.sda3.in‚7.1875 disk.dm-0.out‚ 11532.4140625 disk.dm-1.out‚ 191.640625 network.sent‚ 5.768855e+06 disk.drbd1777.in‚ 1.03515625$ disk.nvme0c0n1.out‚ 11532.4140625 disk.rbd3.out‚0# proc.memory.rssMB‚ 2863.28515625 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda.in‚ 7.23828125% disk./.usageGB‚1098.2088165283203 disk.rbd3.in‚0 disk.rbd1.in‚0) proc.memory.percent‚17.47610569000244) disk./.usagePercent‚73.70171513578804 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.sda.out‚ 191.6796875 disk.sda3.out‚ 191.640625" disk.nvme0n1.out‚ 11532.4140625 disk.dm-1.in‚7.1875 proc.cpu.threads‚17,Oï¥2- _wandbruntime‚2692  _runtime‚2692dˆ™xÇ‚Á _step‚226 & lr-AdamW/pg1‚0.0007607847683394238 & lr-AdamW/pg2‚0.0007607847683394238 trainer/global_step‚2799 % _timestamp‚1.7884192220363276e+09 _runtime‚2692.786243913£!ZÌƒÆ & lr-AdamW/pg1‚0.0007607847683394238 & lr-AdamW/pg2‚0.0007607847683394238 trainer/global_step‚2799 % _timestamp‚1.7884192220363276e+09 _runtime‚2692.786243913 _step‚226â°ÖDz2„- _wandbruntime‚2692  _runtime‚2692©ÃW”Ú'…ÔO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72790690 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104778957042688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118743931279360 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115851800979456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124272310 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124297981 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30134736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124033504197632 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115852047185408 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48958999 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104778974082048 _step‚227 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5194999240192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788419222037021e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5194999240192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124033811639808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 _runtime‚2692.787156382 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚110657047945216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51507289 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51507291 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115851800979456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚110657029942272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307442176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚110657029942272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72765021 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5289856420352 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5179697265609 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72790690 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48958345 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104778957042688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118743931279360 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115851800979456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124272310 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMoni‰¨q…âtor.on_train_batch_start/allocation.all.allocated‚ 124297981 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30134736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124033504197632 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115852047185408 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48958999 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104778974082048 _step‚227 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5194999240192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788419222037021e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5194999240192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124033811639808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 _runtime‚2692.787156382 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚110657047945216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51507289 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51507291 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115851800979456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚110657029942272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307442176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚110657029942272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72765021 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5289856420352 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5179697265609 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72790690 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48958345 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5196836947968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚110695367886336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30145089 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51525010 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5196608731648 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚110695367886336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5196608731648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51525008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118785124915712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51525008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72816267 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115892222837248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚110695385889280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124076536259072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51525010 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5196836947968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115892222837248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104815259500544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5181529665209 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5181753314561 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124341277 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 _runtime‚ 2693.6421314 _step‚228 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18830883 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118785100975616 trainer/global_step‚2799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72816267 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5291714577920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104815276539904 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124341277 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72790572 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48975316 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18830227 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5291435283456 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884192228915274e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48975974 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72790572 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚110695385889280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚109997029854465 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294464^+!è&‰âM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚110695385889280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124076536259072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51525010 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5196836947968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115892222837248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104815259500544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5181529665209 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5181753314561 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124341277 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 _runtime‚ 2693.6421314 _step‚228 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18830883 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118785100975616 trainer/global_step‚2799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72816267 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5291714577920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104815276539904 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124341277 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72790572 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48975316 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18830227 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5291435283456 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884192228915274e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48975974 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72790572 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚110695385889280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚109997029854465 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚109996789165753 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5196836947968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚110695367886336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30145089 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51525010 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5196608731648 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚110695367886336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5196608731648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51525008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118785124915712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51525008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72816267 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115892222837248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299ä–^ñ\2Š- _wandbruntime‚2693  _runtime‚26933Q¤ÝT‹Î 2loss/SRqqt3/assignment_loss‚1.8538823402395435 /loss/FBt1/assignment_loss‚0.670447838306427 .loss/FRt2/detection_loss‚1.834527859153847 /loss/FRt4/detection_loss‚2.6536564884647245 _runtime‚2693.643210693 /loss/FRt2/assignment_loss‚2.194993124653896 /loss/FBt2/assignment_loss‚0.267435870036727 1loss/SRqqt2/detection_loss‚1.2320317304382722 .loss/FBt1/detection_loss‚1.899073725938797 0loss/FRt3/assignment_loss‚3.3756162410086774 /loss/FRt4/assignment_loss‚4.295426303340543 2loss/SRqqt2/assignment_loss‚1.2802900917828084 .loss/FRt3/detection_loss‚2.431865926752699 1loss/SRqqt3/detection_loss‚1.9160978203895045 /loss/FBt2/detection_loss‚0.5027519329092823 .loss/FBt3/detection_loss‚3.858078718185425 /loss/FRt1/assignment_loss‚4.198239731788635 2loss/SRqqt1/assignment_loss‚2.3770730535189313 1loss/FBt3/assignment_loss‚0.08590821176767349 1loss/SRqqt1/detection_loss‚2.7135857462882997 2loss/SRqqt4/assignment_loss‚2.1621985776970782 0loss/FBt4/assignment_loss‚0.7551257876413209 % _timestamp‚1.7884192228933978e+09 _step‚229 .loss/FRt1/detection_loss‚2.273383135907352 epoch‚2 trainer/global_step‚2799 1loss/SRqqt4/detection_loss‚2.1547327941904464 -loss/FBt4/detection_loss‚2.31900760105678 %loss/total_loss‚49.30543065145769q#µYŒÓ 2loss/SRqqt4/assignment_loss‚2.1621985776970782 0loss/FBt4/assignment_loss‚0.7551257876413209 % _timestamp‚1.7884192228933978e+09 _step‚229 .loss/FRt1/detection_loss‚2.273383135907352 epoch‚2 trainer/global_step‚2799 1loss/SRqqt4/detection_loss‚2.1547327941904464 -loss/FBt4/detection_loss‚2.31900760105678 %loss/total_loss‚49.30543065145769 2loss/SRqqt3/assignment_loss‚1.8538823402395435 /loss/FBt1/assignment_loss‚0.670447838306427 .loss/FRt2/detection_loss‚1.834527859153847 /loss/FRt4/detection_loss‚2.6536564884647245 _runtime‚2693.643210693 /loss/FRt2/assignment_loss‚2.194993124653896 /loss/FBt2/assignment_loss‚0.267435870036727 1loss/SRqqt2/detection_loss‚1.2320317304382722 .loss/FBt1/detection_loss‚1.899073725938797 0loss/FRt3/assignment_loss‚3.3756162410086774 /loss/FRt4/assignment_loss‚4.295426303340543 2loss/SRqqt2/assignment_loss‚1.2802900917828084 .loss/FRt3/detection_loss‚2.431865926752699 1loss/SRqqt3/detection_loss‚1.9160978203895045 /loss/FBt2/detection_loss‚0.5027519329092823 .loss/FBt3/detection_loss‚3.858078718185425 /loss/FRt1/assignment_loss‚4.198239731788635 2loss/SRqqt1/assignment_loss‚2.3770730535189313 1loss/FBt3/assignment_loss‚0.08590821176767349 1loss/SRqqt1/detection_loss‚2.7135857462882997åÄÝñЂ(:„ ¹äÔÀý•x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚97.402' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.4804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L0OsŽ‚(:è ¹äÔ­þ»z! disk.rbd2.out‚118.99169921875 disk.rbd0.in‚0) proc.memory.percent‚17.47610569000244 disk.sda2.out‚ 0.0390625& disk.nvme0n1.out‚11631.56689453125 disk.rbd3.out‚0( disk./.usagePercent‚73.7017164158257 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.rbd1.out‚ 1.28955078125 disk.nvme0n1.in‚ 4.9296875 network.sent‚ 5.775237e+06 disk.sda.out‚ 192.109375 disk.dm-0.in‚ 4.9296875 disk.sda2.in‚0 disk.dm-1.in‚7.1875 disk.rbd2.in‚0$ memory_percent‚98.94247055053711& proc.memory.availableMB‚ 173.265625% disk./.usageGB‚1098.2088356018066 disk.rbd1.in‚0 disk.dm-1.out‚ 192.0703125 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.23828125# disk.dm-0.out‚11631.56689453125 disk.rbd4.out‚0 disk.sda3.in‚7.1875 disk.rbd3.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2863.28515625 cpu‚25.26850902229333 disk.sda1.out‚0 network.recv‚960054 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚11631.56689453125 disk.sda3.out‚ 192.0703125ÇÎ,š‚(: ¬¹äÔ–Ú¬x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.08' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.215999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e/ëz‚(:ï ¬¹äÔí†Ü{ network.recv‚965883 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 192.72265625# disk.dm-0.out‚11730.66064453125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9296875 disk.dm-1.in‚7.3125 disk.rbd2.in‚0# proc.memory.rssMB‚ 2863.28515625) disk./.usagePercent‚73.70171846388598 disk.sda3.in‚7.3125& disk.nvme0n1.out‚11730.66064453125 disk.rbd4.in‚0( proc.memory.availableMB‚ 173.24609375 proc.cpu.threads‚17 disk.sda3.out‚ 192.68359375 disk.rbd1.out‚ 1.28955078125( disk.nvme0c0n1.out‚11730.66064453125 disk.sda1.out‚0% disk./.usageGB‚1098.2088661193848 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0$ memory_percent‚98.94258975982666 network.sent‚ 5.883599e+06 disk.dm-1.out‚ 192.68359375! disk.rbd2.out‚118.99169921875 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd3.out‚0 cpu‚25.26728962770858 disk.sda.in‚ 7.36328125 disk.dm-0.in‚ 4.9296875 disk.rbd0.in‚0) proc.memory.percent‚17.47610569000244Þ%G‘‚(:„ »¹äÔÉߘx gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|ïéYx’‚(:í »¹äÔþ°ô{ disk.nvme0n1.in‚ 4.9296875 disk.sda2.out‚ 0.0390625# proc.memory.rssMB‚ 2863.28515625 cpu‚25.266103526585773& disk.nvme0n1.out‚11830.19873046875% disk./.usageGB‚1098.2088928222656( disk.nvme0c0n1.out‚11830.19873046875 disk.rbd1.out‚ 1.28955078125 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚ 119.16015625 disk.drbd1777.out‚0 disk.rbd4.out‚0# disk.dm-0.out‚11830.19873046875 disk.sda2.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 173.24609375 network.recv‚970210 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 4.9296875$ memory_percent‚98.94258975982666) proc.memory.percent‚17.47610569000244 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.rbd1.in‚0 disk.dm-1.in‚7.3125 disk.nvme0c0n1.in‚ 4.9296875 network.sent‚ 5.889702e+06) disk./.usagePercent‚73.70172025593872 disk.sda.in‚ 7.36328125 disk.sda.out‚ 193.04296875 disk.sda3.out‚ 193.00390625 disk.dm-1.out‚ 193.00390625ÐBŽ2“- _wandbruntime‚2737  _runtime‚2737þø¥9Š”Á _runtime‚2737.853975734 _step‚230 & lr-AdamW/pg1‚0.0007581812270698483 & lr-AdamW/pg2‚0.0007581812270698483 trai?´¦2=ner/global_step‚2849 % _timestamp‚1.7884192671040459e+09½_*3Ì•Æ _step‚230 & lr-AdamW/pg1‚0.0007581812270698483 & lr-AdamW/pg2‚0.0007581812270698483 trainer/global_step‚2849 % _timestamp‚1.7884192671040459e+09 _runtime‚2737.853975734æL˜éQ2–- _wandbruntime‚2737  _runtime‚2737ÃÏ«MÚ'—ÔO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126464833 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74045923 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚117860843525632 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117860597319680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19146032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5383326649856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111865651460809 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52393239 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30652484 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5286898377216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289793536 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5271554485961 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112573927145472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126439162 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120802440032256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 trainer/global_step‚2849 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚117860597319680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5286670174208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126185452948480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111865410772305 _step‚231 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74045923 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112573945148416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30652486 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚112573927145472 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 _runtime‚ 2737.85495588 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74071592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5383036856320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 % _timestamp‚1.7884192671047873e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49797862 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126185766682112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52393241 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117860843525632 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74071592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5286670174208 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5286898377216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313733632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106594079935488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52393239 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0/éa£ß'˜ÙO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74071592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5383036856320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 % _timestamp‚1.7884192671047873e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49797862 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126185766682112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52393241 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117860843525632 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74071592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5286670174208 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5286898377216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313733632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106594079935488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52393239 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126464833 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74045923 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚117860843525632 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117860597319680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19146032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5383326649856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111865651460809 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52393239 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30652484 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5286898377216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289793536 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5271554485961 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112573927145472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126439162 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120802440032256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 trainer/global_step‚2849 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚117860597319680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5286670174208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126185452948480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111865410772305 _step‚231 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74045923 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112573945148416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30652486 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚112573927145472 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 _runtime‚ 2737.85495588çYD2™- _wandbruntime‚2738  _runtime‚2738a…tSå&šßM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49815021 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112612265089536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30662841 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5384920699392 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126482432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52410958 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52410960 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117901019177472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5288736084992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5288736084992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106630399432704 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74071474 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117900772958208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5273163236417 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126228811819520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5273386885769 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19151548 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5288507868672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120843585788416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52410958 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74097169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49814387 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884192679678361e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126482432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2849 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52410960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19152180 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126228506487808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5288507868672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120843609728512 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126508129 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74097169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126508129 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _step‚232 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112612265089536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111903786318473 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceSÿ¦-tatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5288507868672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120843585788416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52410958 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74097169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49814387 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884192679678361e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126482432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2849 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52410960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19152180 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126228506487808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5288507868672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120843609728512 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126508129 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74097169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126508129 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _step‚232 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112612265089536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111903786318473 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5384920699392 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126482432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52410958 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52410960 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256è…ÀïÙ2œ- _wandbruntime‚2738  _runtime‚2738S}UÞC½ /loss/FRt3/detection_loss‚2.6801071344195186 .loss/FRt4/detection_loss‚2.084136442935213 %loss/total_loss‚46.28116569006453 /loss/FBt1/assignment_loss‚0.729027960035536 1loss/FBt2/assignment_loss‚0.28752352013848564 0loss/SRqqt3/detection_loss‚1.999377739650232 2loss/SRqqt2/assignment_loss‚1.4435261538558042 1loss/SRqqt4/detection_loss‚1.7909202730655671 _runtime‚2738.719402997 0loss/FRt4/assignment_loss‚3.6242518970306885 2loss/SRqqt4/assignment_loss‚1.6953853645920753 /loss/FBt2/detection_loss‚0.7788522440548006 $ _timestamp‚1.788419267969582e+09 /loss/FRt1/assignment_loss‚3.959495927012244 /loss/FRt2/detection_loss‚1.7700883707985646 0loss/SRqqt1/detection_loss‚2.564670917391777 0loss/SRqqt2/detection_loss‚1.207817488419476 _step‚233 /loss/FRt2/assignment_loss‚2.106980851510676 .loss/FBt4/detection_loss‚2.129373073577881 epoch‚2 0loss/FRt3/assignment_loss‚3.8914614948066504 1loss/SRqqt3/assignment_loss‚1.768419574532244 .loss/FRt1/detection_loss‚2.614688772101735 trainer/global_step‚2849 2loss/SRqqt1/assignment_loss‚2.2627033869425457 loss/FBt3/assignment_loss‚0 0loss/FBt4/assignment_loss‚0.4817197079433494 .loss/FBt1/detection_loss‚2.224486963616477 /loss/FBt3/detection_loss‚2.1861504316329956”ÝužHž /loss/FRt2/detection_loss‚1.7700883707985646 0loss/SRqqt1/detection_loss‚2.564670917391777 0loss/SRqqt2/detection_loss‚1.207817488419476 _step‚233 /loss/FRt2/assignment_loss‚2.106980851510676 .loss/FBt4/detection_loss‚2.129373073577881 epoch‚2 0loss/FRt3/assignment_loss‚3.8914614948066504 1loss/SRqqt3/assignment_loss‚1.768419574532244 .loss/FRt1/detection_loss‚2.614688772101735 trainer/global_step‚2849 2loss/SRqqt1/assignment_loss‚2.2627033869425457 loss/FBt3/assignment_loss‚0 0loss/FBt4/assignment_loss‚0.4817197079433494 .loss/FBt1/detection_loss‚2.224486963616477 /loss/FBt3/detection_loss‚2.1861504316329956 /loss/FRt3/detection_loss‚2.6801071344195186 .loss/FRt4/detection_loss‚2.084136442935213 %loss/total_loss‚46.28116569006453 /loss/FBt1/assignment_loss‚0.729027960035536 1loss/FBt2/assignment_loss‚0.28752352013848564 0loss/SRqqt3/detection_loss‚1.999377739650232 2loss/SRqqt2/assignment_loss‚1.4435261538558042 1loss/SRqqt4/detection_loss‚1.7909202730655671 _runtime‚2738.719402997 0loss/FRt4/assignment_loss‚3.6242518970306885 2loss/SRqqt4/assignment_loss‚1.6953853645920753 /loss/FBt2/detection_loss‚0.7788522440548006 $ _timestamp‚1.788419267969582e+09 /loss/FRt1/assignment_loss‚3.959495927012244é»ÍŸ‚(:… ʹäÔ¯¨³x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.687' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1374 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š}•g ‚(:Ü ʹäÔ¬¢«{ disk.rbd2.in‚0# proc.memory.rssMB‚ 2863.28515625 proc.cpu.threads‚17 disk.sda.out‚ 193.73046875% proc.memory.availableMB‚ 172.71875) proc.memory.percent‚17.47610569000244 cpu‚25.264944643627075 network.sent‚ 5.8956e+06) disk./.usagePercent‚73.70171513578804 disk.rbd0.in‚0 disk.dm-0.in‚ 4.9296875 disk.sda2.in‚0 disk.rbd2.out‚ 119.16015625 disk.rbd0.out‚0 disk.rbd4.in‚0 network.recv‚974565$ memory_percent‚98.94580841064453% disk./.usageGB‚1098.2088165283203 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚ 4.9296875$ disk.nvme0c0n1.out‚ 11830.7265625 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0" disk.nvme0n1.out‚ 11830.7265625 disk.dm-1.out‚ 193.69140625 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 1.28955078125 disk.nvme0n1.in‚ 4.9296875 disk.dm-0.out‚ 11830.7265625 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 193.69140625 disk.sda2.out‚ 0.0390625o49+¡‚(:… Ù¹äÔúÓÊx gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Ð)Ùl¢‚(:á Ù¹äÔáÜëz disk.sda3.out‚ 194.1328125 disk.rbd1.out‚ 1.28955078125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚11929.97021484375 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0# disk.dm-0.out‚11929.97021484375 disk.rbd2.out‚119.25 network.recv‚981324$ disk./.usageGB‚1098.208164215088( proc.memory.availableMB‚ 172.89453125) proc.memory.percent‚17.47610569000244 network.sent‚ 6.004807e+06) disk./.usagePercent‚73.70167135849968 disk.sda.out‚ 194.171875 disk.drbd1777.in‚ 1.03515625 cpu‚25.26370724481154 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 4.9296875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 194.1328125 disk.rbd2.in‚0 disk.sda3.in‚7.3125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2863.28515625 disk.rbd4.in‚0& disk.nvme0n1.out‚11929.97021484375 disk.dm-0.in‚ 4.9296875$ memory_percent‚98.94473552703857 disk.nvme0c0n1.in‚ 4.9296875i˜Ø‹£‚(:… è¹äÔÆ…x gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©-Ev¤‚(:ë è¹äÔµô¶z) disk./.usagePercent‚73.70167135849968 disk.nvme0n1.in‚ 4.9296875) proc.memory.percent‚17.47610569000244$ disk./.usageGB‚1098.208164215088 disk.sda2.in‚0( disk.nvme0c0n1.out‚12029.05615234375 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.dm-0.in‚ 4.9296875 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚98.94492626190186# proc.memory.rssMB‚ 2863.28515625 network.sent‚ 6.010067e+06 disk.rbd1.out‚ 1.28955078125 disk.dm-1.out‚ 194.64453125 disk.rbd4.out‚0 disk.sda3.out‚ 194.64453125( proc.memory.availableMB‚ 172.86328125 network.recv‚984589 disk.rbd1.in‚0# disk.dm-0.out‚12029.05615234375 disk.rbd2.out‚ 120.76171875 disk.nvme0c0n1.in‚ 4.9296875 disk.sda.out‚ 194.68359375 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 cpu‚25.26257063667072 proc.cpu.threads‚17 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚12029.05615234375 disk.sda2.out‚ 0.0390625 disk.rbd3.in‚00û92¥- _wandbruntime‚2782  _runtime‚2782[¿BÔƦÀ trainer/global_step‚2899 $ _timestamp‚1.788419312067119e+09 _runtime‚2782.817057828 _step‚234 & lr-AdamW/pg1‚0.0007555032829731927 & lr-AdamW/pg2‚0.0007555032829731927 “þÙ˧Š_step‚234 & lr-AdamW/pg1‚0.0007555032829731927 & lr-AdamW/pg2‚0.0007555032829731927 trainer/global_step‚2899 $ _timestamp‚1.788419312067119e+09 _runtime‚2782.817057828ê²uñ2¨- _wandbruntime‚2782  _runtime‚2782FÇyÛ'©ÕO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5476779185152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113772407924817 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31170234 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 % _timestamp‚1.7884193120678663e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128606014 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53279191 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128631685 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53279191 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5378797514240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19467886 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53279189 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119869639865856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114490824348672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108409202828288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5362964408025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19468534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5363188057169 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5378569311232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122860924845056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113772167236313 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50638120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚128337704030208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75352494 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119869393659904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50638770 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119869639865856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119869393659904 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53279189 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75326825 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944úÏ!ݪÚO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75352494 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119869393659904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50638770 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119869639865856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119869393659904 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53279189 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75326825 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5476779185152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113772407924817 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31170234 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 % _timestamp‚1.7884193120678663e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128606014 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53279191 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128631685 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53279191 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5378797514240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.élB'freed‚19467886 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53279189 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119869639865856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114490824348672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108409202828288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5362964408025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19468534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5363188057169 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5378569311232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122860924845056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113772167236313 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50638120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚128337704030208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128ësŒ©2«- _wandbruntime‚2783  _runtime‚2783‘)Ù1å&¬ßM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚128380745071616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119909569298432 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128674981 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚236 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53296908 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113810302093769 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114529180295680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5365020456977 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75378071 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275100160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75378071 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299040256 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114529180295680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚114529162292736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113810542782481 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2783.672265584 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75352376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19474052 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122902094541312 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108445505286144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5364796807625 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119909815517696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50654641 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19474692 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122902070601216 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 äpê&­äM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128380446031360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5478375430144 % _timestamp‚1.7884193129217076e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75352376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119909815517696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31180591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128649284 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108445522325504 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50655283 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53296910 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128649284 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114529180295680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚114529162292736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113810542782481 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2783.672265584 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75352376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19474052 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122902094541312 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108445505286144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5364796807625 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119909815517696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50654641 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19474692 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122902070601216 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚128380745071616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119909569298432 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128674981 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚236 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53296908 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113810302093769 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114529180295680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5365020456977 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75378071 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275100160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75378071 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299040256 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54165139 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76607727 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124919409657856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚2828.045758643 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚116407739554816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121878190000128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚644 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚110224325721088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884193572955966e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚121878436206080 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚121878190000128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5454821628377 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121878436206080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31687986 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19792890 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5569863221760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116407721551872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268822016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76607727 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124919385717760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115679164388825 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76633396 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19793534 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115678923700321 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130798537 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54165141 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5470696651264 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5454597979233 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚646 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31687984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5470696651264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 trainer/global_step‚2949 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51481520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5570132043776 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚116407739554816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268822016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76607727 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124919385717760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115679164388825 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76633396 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19793534 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115678923700321 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130798537 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54165141 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5470696651264 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5454597979233 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚646 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31687984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5470696651264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 trainer/global_step‚2949 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51481520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5570132043776 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚116407739554816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54165139 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76607727 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124919409657856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚2828.045758643 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚116407739554816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121878190000128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚644 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚110224325721088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884193572955966e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚121878436206080 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚121878190000128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5454821628377 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121878436206080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31687986 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19792890 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5569863221760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624ï¼¾í2½- _wandbruntime‚2828  _runtime‚2828ehmã&¾ÝM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚110260645218304 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121918611857920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54182860 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2949 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19800082 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121918365638656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚116446077498880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110260628178944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116446077498880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5571734785536 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121918611857920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115717299246489 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5472534359040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130841833 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651648 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115717058557777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130816136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124960555414016 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130816136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76633278 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5572001497088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51497786 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130816136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76633278 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5572001497088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51497786 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121918611857920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54182860 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_t£J².rain_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2949 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19800082 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121918365638656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚116446077498880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110260628178944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116446077498880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5571734785536 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121918611857920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115717299246489 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5472534359040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130841833 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651648 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115717058557777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130816136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124960555414016 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5546231550441 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132641463954432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788419402513826e+09 _step‚243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132641754619392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118324636758016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118324618755072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20116723 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123886986340352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123887232546304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5562595788288 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55051091 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5562595788288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55051091 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112039465653248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77888629 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123886986340352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52321818 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5663860148736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55051089 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32205734 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664960 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5663593423872 trainer/global_step‚2999 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5663593423872 trainer/global_step‚2999 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5546231550441 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132641463954432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788419402513826e+09 _step‚243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132641754619392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118324636758016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118324618755072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20116723 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123886986340352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123887232546304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5562595788288 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55051091 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5562595788288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55051091 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112039465653248 L?DeviceStatsMonitor.on_traia=GÞan_batch_start/active.small_pool.freed‚77888629 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123886986340352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52321818 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5663860148736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55051089 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32205734 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664960 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 _runtime‚2874.128738224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20123404 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 305331712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132684491237376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117624055710497 trainer/global_step‚2999 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚118362956699136 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5564433496064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55068810 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77939875 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112075751071744 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77939875 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32216091 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5548287599393 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚123927161978880 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚118362974702080 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5564433496064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5665732402176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55068810 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127019064166912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5564205279744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123927408198144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77914180 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5665451010560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132982988 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 _step‚244 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133008685 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5548063950041 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55068808 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52339495 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚118362974702080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32216089 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117623815021785 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112075768111104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55068808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52338865 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5564205279744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133008685 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132982988 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264…cëýê&ÑäM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52339495 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚118362974702080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32216089 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117623815021785 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112075768111104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55068808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52338865 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5564205279744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133008685 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132982988 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 _runtime‚2874.128738224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20123404 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 305331712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132684491237376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117624055710497 trainer/global_step‚2999 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚118362956699136 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5564433496064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55068810 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77939875 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112075751071744 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77939875 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32216091 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5548287599393 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚123927161978880 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚118362974702080 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5564433496064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5665732402176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55068810 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127019064166912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5564205279744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123927408198144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77914180 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5665451010560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132982988 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 _step‚244 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133008685 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5548063950041 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55068808 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624ôF×…™2Ò- _wandbruntime‚2874  _runtime‚2874ìÊ ¡XÓÒ .loss/FBt4/detection_loss‚2.202212957235483 &loss/total_loss‚50.997712714048994 % _timestamp‚1.7884194033799903e+09 -loss/FRt3/assignment_loss‚3.2211458099925 0loss/FBt1/assignment_loss‚0.6758417906193586 1loss/SRqqt2/detection_loss‚1.2245160327071236 1loss/SRqqt4/detection_loss‚2.0930762500143967 .loss/FBt1/detection_loss‚2.159505493938923 2loss/SRqqt4/assignment_loss‚2.3003052037495832 /loss/FRt2/detection_loss‚1.6548456896324546 /loss/FRt4/detection_loss‚2.6034775486117914 epoch‚2 0loss/FRt2/assignment_loss‚2.7901479553203195 2loss/SRqqt2/assignment_loss‚1.3439887228702743 1loss/FBt3/assignment_loss‚0.09678877597131456 0loss/SRqqt1/detection_loss‚2.741902942103999 1loss/SRqqt3/detection_loss‚1.8768769853093006 .loss/FBt3/detection_loss‚5.180703798929851 _step‚245 1loss/SRqqt1/assignment_loss‚2.231529346534184 0loss/FBt4/assignment_loss‚0.5145976436772168 /loss/FBt2/detection_loss‚0.5315527959285598 1loss/FBt2/assignment_loss‚0.23974076208711645 /loss/FRt3/detection_loss‚2.4908656257650126 0loss/FRt1/assignment_loss‚4.1876527902327085 /loss/FRt4/assignment_loss‚4.003556006833127 2loss/SRqqt3/assignment_loss‚1.9165354475930885 trainer/global_step‚2999 _runtime‚2874.129822487 .loss/FRt1/detection_loss‚2.716346338391304 ]Ô× 0loss/FRt1/assignment_loss‚4.1876527902327085 /loss/FRt4/assignment_loss‚4.003556006833127 2loss/SRqqt3/assignment_loss‚1.9165354475930885 trainer/global_step‚2999 _runtime‚2874.129822487 .loss/FRt1/detection_loss‚2.716346338391304 .loss/FBt4/detection_loss‚2.202212957235483 &loss/total_loss‚50.997712714048994 % _timestamp‚1.7884194033799903e+09 -loss/FRt3/assignment_loss‚3.2211458099925 0loss/FBt1/assignment_loss‚0.6758417906193586 1loss/SRqqt2/detection_loss‚1.2245160327071236 1loss/SRqqt4/detection_loss‚2.0930762500143967 .loss/FBt1/detection_loss‚2.159505493938923 2loss/SRqqt4/assignment_loss‚2.3003052037495832 /loss/FRt2/detection_loss‚1.6548456896324546 /loss/FRt4/detection_loss‚2.6034775486117914 epoch‚2 0loss/FRt2/assignment_loss‚2.7901479553203195 2loss/SRqqt2/assignment_loss‚1.3439887228702743 1loss/FBt3/assignment_loss‚0.09678877597131456 0loss/SRqqt1/detection_loss‚2.741902942103999 1loss/SRqqt3/detection_loss‚1.8768769853093006 .loss/FBt3/detection_loss‚5.180703798929851 _step‚245 1loss/SRqqt1/assignment_loss‚2.231529346534184 0loss/FBt4/assignment_loss‚0.5145976436772168 /loss/FBt2/detection_loss‚0.5315527959285598 1loss/FBt2/assignment_loss‚0.23974076208711645 /loss/FRt3/detection_loss‚2.4908656257650126õ žØ›Õ‚(: ѺäÔ¶˜«x gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.356' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.871199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îYßroÖ‚(:ä ѺäÔí‹{# memory_percent‚98.9436149597168# proc.memory.rssMB‚ 2863.28515625) proc.memory.percent‚17.47610569000244% disk./.usageGB‚1098.1752014160156 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd1.out‚ 1.28955078125& proc.memory.availableMB‚ 173.078125 disk.dm-0.out‚ 12527.484375 disk.rbd4.in‚0 disk.sda3.out‚ 198.37109375 network.recv‚ 1.018457e+06 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 198.37109375 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0# disk.nvme0c0n1.out‚ 12527.484375 disk.sda1.out‚0! disk.rbd2.out‚122.66064453125 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 4.9296875! disk.nvme0n1.out‚ 12527.484375 cpu‚25.254706686289005 proc.cpu.threads‚17 disk.dm-0.in‚ 4.9296875 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 198.41015625 disk.rbd2.in‚0 disk.rbd0.out‚0 network.sent‚ 6.256768e+06) disk./.usagePercent‚73.69945919739631 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125Ä­Ž×‚(:ƒ àºäÔÖ¤²x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.295' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.659 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É©XË{Ø‚(:ð àºäÔãéÅz disk.dm-1.in‚7.3125 network.recv‚ 1.024149e+06 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.dm-1.out‚ 198.859375) disk./.usagePercent‚73.69945919739631 disk.rbd1.in‚0 disk.rbd1.out‚ 1.28955078125 disk.rbd3.in‚0( proc.memory.availableMB‚ 173.07421875 disk.drbd1777.out‚0 disk.sda.out‚ 198.8984375 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.out‚0 disk.rbd3.out‚0# disk.dm-0.out‚12626.83642578125 disk.rbd0.out‚0# memory_percent‚98.9436388015747 cpu‚25.253580448236363 network.sent‚ 6.364942e+06 disk.nvme0n1.in‚ 4.9296875& disk.nvme0n1.out‚12626.83642578125 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 proc.cpu.threads‚17 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625( disk.nvme0c0n1.out‚12626.83642578125 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.sda3.out‚ 198.859375# proc.memory.rssMB‚ 2863.28515625% disk./.usageGB‚1098.1752014160156! disk.rbd2.out‚122.67626953125) proc.memory.percent‚17.47610569000244WdnTšÙ‚(: ïºäÔÆ€›x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.496' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.899199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iE;7}Ú‚(:ò ïºäÔŒîÈ{ disk.nvme0c0n1.in‚ 4.9296875( proc.memory.availableMB‚ 172.82421875 network.recv‚ 1.028547e+06 disk.sda3.in‚7.3125# disk.dm-0.out‚12726.22998046875 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚12726.22998046875 disk.rbd3.out‚0 cpu‚25.252522487609 disk.rbd3.in‚0$ memory_percent‚98.94516468048096 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda1.out‚0 network.sent‚ 6.370966e+06 disk.dm-0.in‚ 4.9296875 disk.dm-1.out‚ 199.28515625 disk.rbd2.in‚0 disk.sda1.in‚ 0.05078125% disk./.usageGB‚1098.1752243041992 disk.dm-1.in‚7.3125 disk.rbd2.out‚122.8291015625 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 199.32421875 disk.rbd1.out‚ 1.28955078125& disk.nvme0n1.out‚12726.22998046875# proc.memory.rssMB‚ 2863.28515625) proc.memory.percent‚17.47610569000244) disk./.usagePercent‚73.69946073344151 disk.nvme0n1.in‚ 4.9296875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 199.28515625 disk.rbd1.in‚0DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32723486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296956416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79169531 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20439530 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129036379283456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_bat”0üy½ch_start/inactive_split.large_pool.freed‚32723484 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚679 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚125896028886528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134793736401920 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5654494925312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125895782680576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚681 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 trainer/global_step‚3049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚120241533961216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135132241 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53163695 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113854588546048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55937041 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134793439445504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55937039 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55937039 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 % _timestamp‚1.7884194479518461e+09 _runtime‚2918.702053113 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5654266722304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119492436628337 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53163014 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79195200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5757084102144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79195200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273016320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5638088770793 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5654266722304 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120241515958272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125895782680576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360veºýà'àÚO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32723486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296956416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79169531 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20439530 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129036379283456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32723484 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚679 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚125896028886528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134793736401920 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5654494925312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125895782680576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚681 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 trainer/global_step‚3049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚120241533961216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135132241 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53163695 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113854588546048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55937041 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134793439445504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55937039 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55937039 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 % _timestamp‚1.7884194479518461e+09 _runtime‚2918.702053113 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5654266722304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119492436628337 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53163014 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79195200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5757084102144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79195200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273016320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5638088770793 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5654266722304 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120241515958272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125895782680576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113854571506688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55937041 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚125896028886528 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32733839 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20447001 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135149840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53180842 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119530812174505 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120279853902336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 _step‚248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5639697521249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125936204538368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53180162 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119530571485793 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5656332633088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚125936204538368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚2919.563072746 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135175537 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301137408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5758946565120 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120279871905280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134836471604736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113890873964544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5759223762432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79220777 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32733839 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20447001 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135149840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53180842 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119530812174505 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120279853902336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 _step‚248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5639697521249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125936204538368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53180162 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119530571485793 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5656332633088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚125936204538368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚2919.563072746 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135175537 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301137408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5758946565120 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120279871905280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134836471604736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113890873964544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5759223762432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79220777 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127904579020800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137299093 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127904579020800 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5746165859328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56822991 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80476102 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80450433 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127904579020800 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5746165859328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56822991 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80476102 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80450433 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127904579020800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137299093ûñ˜~2ó- _wandbruntime‚2965  _runtime‚2965sÌcå&ôßM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80475984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5748003553792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5748003553792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5852406288896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884194943073218e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20770095 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5748231770112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33251591 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115705996857344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80475984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121437327949801 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56840710 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296943104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131136009852416 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137342389 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20769425 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5852679291904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56840710 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137316692 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33251589 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136988713084416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136988416141312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54021014 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚122196751105536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80501679 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚122196751105536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚122196769108480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137342389 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54021686 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264Lùpê&õäM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127944754659328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115705996857344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80475984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121437327949801 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56840710 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296943104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131136009852416 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137342389 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20769425 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5852679291904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56840710 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137316692 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33251589 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136988713084416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136988416141312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54021014 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚122196751105536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80501679 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚122196751105536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚122196769108480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137342389 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54021686 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80475984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5748003553792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5748003553792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5852406288896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884194943073218e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20770095 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5748231770112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33251591 FI2‚- _wandbruntime‚3009  _runtime‚3009½ÍRcƒÕO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139465945 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21088064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81757004 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57708939 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚124075310364672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117484817292288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5821132264065 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33758986 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884195388487236e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57708941 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129913621566976 @8®q ì\DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5838064996352 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129913375361024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129913375361024 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57708939 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54847050 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33758984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚668 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚123305949556353 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚124075328367616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81731335 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 trainer/global_step‚3149 _step‚255 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54846380 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5821355913209 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3009.598794557 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57708941 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21087396 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚124075310364672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5838064996352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307776 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33758986 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884195388487236e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57708941 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129913621566976 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5838064996352 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129913375361024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129913375361024 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57708939 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54847050 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33758984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚668 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚123305949556353 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚124075328367616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81731335 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 trainer/global_step‚3149 _step‚255 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54846380 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5821355913209 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3009.598794557 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57708941 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824ÿ"*õT2…- _wandbruntime‚3010  _runtime‚3010ÉPl»ä&†ÞM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57726658 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139483544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5945855433728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚124113666311680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57726660 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21093577 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54862916 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139509241 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21094215 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117521136789504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5822964663665 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81782581 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚139140644944896 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57726660 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81782581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33769339 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5840130907136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57726658 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133194494665216Ô®ÿù‹‡ãM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57726658 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139483544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5945855433728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚124113666311680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57726660 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21093577 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54862916 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139509241 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21094215 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117521136789504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5822964663665 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81782581 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚139140644944896 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57726660 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81782581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33769339 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5840130907136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57726658 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133194494665216€ Þ¹*2ˆ- _wandbruntime‚3010  _runtime‚3010XW®T‰Î 0loss/FRt2/assignment_loss‚1.7915766388177872 1loss/SRqqt2/assignment_loss‚1.474433798285631 1loss/SRqqt4/detection_loss‚1.8678318393727145 .loss/FBt1/detection_loss‚2.570523202419281 _step‚257 epoch‚2 trainer/global_step‚3149 .loss/FRt1/detection_loss‚2.533500531461181 0loss/SRqqt1/detection_loss‚2.828952999026687 .loss/FBt4/detection_loss‚2.020440273814731 0loss/FRt3/assignment_loss‚3.4589559983234017 0loss/FRt4/assignment_loss‚3.3735206156242183 .loss/FRt3/detection_loss‚2.376402645694966 1loss/SRqqt3/detection_loss‚1.9128694602271967 0loss/FBt2/detection_loss‚0.47225552846442215 /loss/FRt1/assignment_loss‚4.039874614738837 1loss/SRqqt1/assignment_loss‚2.451922591085787 .loss/FRt2/detection_loss‚2.098332601040602 _runtime‚3010.456115375 1loss/SRqqt4/assignment_loss‚1.903225273390611 /loss/FBt3/detection_loss‚3.4376034259796144 1loss/SRqqt3/assignment_loss‚2.079156662288465 1loss/FBt2/assignment_loss‚0.17398192579568345 1loss/FBt3/assignment_loss‚0.03280470160243567 1loss/SRqqt2/detection_loss‚1.1512752507741635 #loss/total_loss‚47.707041119853 0loss/FBt1/assignment_loss‚0.5045859568009081 0loss/FBt4/assignment_loss‚0.7882546802413546 % _timestamp‚1.7884195397062724e+09 .loss/FRt4/detection_loss‚2.364759904582326í ×YŠÓ .loss/FRt3/detection_loss‚2.376402645694966 1loss/SRqqt3/detection_loss‚1.9128694602271967 0loss/FBt2/detection_loss‚0.47225552846442215 /loss/FRt1/assignment_loss‚4.039874614738837 1loss/SRqqt1/assignment_loss‚2.451922591085787 .loss/FRt2/detection_loss‚2.098332601040602 _runtime‚3010.456115375 1loss/SRqqt4/assignment_loss‚1.903225273390611 /loss/FBt3/detection_loss‚3.4376034259796144 1loss/SRqqt3/assignment_loss‚2.079156662288465 1loss/FBt2/assignment_loss‚0.17398192579568345 1loss/FBt3/assignment_loss‚0.03280470160243567 1loss/SRqqt2/detection_loss‚1.1512752507741635 #loss/total_loss‚47.707041119853 0loss/FBt1/assignment_loss‚0.5045859568009081 0loss/FBt4/assignment_loss‚0.7882546802413546 % _timestamp‚1.7884195397062724e+09 .loss/FRt4/detection_loss‚2.364759904582326 0loss/FRt2/assignment_loss‚1.7915766388177872 1loss/SRqqt2/assignment_loss‚1.474433798285631 1loss/SRqqt4/detection_loss‚1.8678318393727145 .loss/FBt1/detection_loss‚2.570523202419281 _step‚257 epoch‚2 trainer/global_step‚3149 .loss/FRt1/detection_loss‚2.533500531461181 0loss/SRqqt1/detection_loss‚2.828952999026687 .loss/FBt4/detection_loss‚2.020440273814731 0loss/FRt3/assignment_loss‚3.4589559983234017 0loss/FRt4/assignment_loss‚3.3735206156242183ô¹Øü‹‚(:„ Ø»äÔÕ–šx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Pn+sŒ‚(:è Ø»äÔÙ¨Þz disk.dm-1.in‚7.3125! disk.rbd2.out‚126.25146484375# proc.memory.rssMB‚ 2863.28515625 disk.sda1.in‚ 0.05078125) disk./.usagePercent‚73.62819898822546 disk.rbd0.in‚0$ memory_percent‚98.94626140594482( proc.memory.availableMB‚ 172.64453125 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚203.5 disk.sda3.in‚7.3125 disk.rbd1.out‚ 1.533203125 network.recv‚ 1.062445e+06' disk.nvme0c0n1.out‚12994.8056640625 disk.rbd1.in‚0% disk.nvme0n1.out‚12994.8056640625 disk.rbd2.in‚0) proc.memory.percent‚17.47610569000244" disk.dm-0.out‚12994.8056640625 disk.sda2.out‚ 0.0390625 network.sent‚ 6.618349e+06 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 4.9296875 disk.dm-0.in‚ 4.9296875 disk.dm-1.out‚ 203.4609375% disk./.usageGB‚1097.1133728027344 disk.sda3.out‚ 203.4609375 disk.sda2.in‚0 cpu‚25.245462766132842 proc.cpu.threads‚17 disk.rbd4.out‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 4.9296875” ñ›‚(: ç»äÔ¸º¡x gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.225' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.244999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓÄÁøxŽ‚(:í ç»äÔ‰ˆÞ{ disk.dm-1.out‚ 203.984375 disk.sda.out‚ 204.0234375 disk.sda3.in‚7.3125 disk.sda3.out‚ 203.984375 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 network.recv‚ 1.069674e+06) disk./.usagePercent‚73.62819642815012 disk.sda1.out‚0 disk.sda2.in‚0! disk.rbd2.out‚126.37646484375# proc.memory.rssMB‚ 2863.28515625) proc.memory.percent‚17.47610569000244% disk./.usageGB‚1097.1133346557617 disk.rbd4.out‚0 network.sent‚ 6.727579e+06 disk.dm-1.in‚7.3125 cpu‚25.24440474576381( disk.nvme0c0n1.out‚12995.18115234375 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚ 4.9296875 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.dm-0.in‚ 4.9296875 disk.rbd2.in‚0# memory_percent‚98.9447832107544( proc.memory.availableMB‚ 172.88671875 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 1.533203125& disk.nvme0n1.out‚12995.18115234375# disk.dm-0.out‚12995.18115234375 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9296875]ŠÉõ‚(:„ ö»äÔ ê®x gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s4q|‚(:ñ ö»äÔ좇| network.recv‚ 1.073405e+06 disk.sda2.in‚0 disk.rbd1.out‚ 1.59619140625 disk.dm-1.out‚ 204.6015625 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd1.in‚0' proc.memory.availableMB‚ 172.3671875) proc.memory.percent‚17.47610569000244 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 cpu‚25.243368368497922 network.sent‚ 6.733542e+06 disk.nvme0n1.in‚ 4.9296875 disk.dm-0.in‚ 4.9296875 disk.rbd2.in‚0 disk.drbd1777.out‚0# disk.dm-0.out‚12995.61962890625( disk.nvme0c0n1.out‚12995.61962890625$ memory_percent‚98.94795417785645# proc.memory.rssMB‚ 2863.28515625 proc.cpu.threads‚17! disk.rbd2.out‚126.39208984375 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125) disk./.usagePercent‚73.62819642815012% disk./.usageGB‚1097.1133346557617& disk.nvme0n1.out‚12995.61962890625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda3.in‚7.3125 disk.sda3.out‚ 204.6015625 disk.rbd3.out‚0 disk.sda.out‚ 204.640625/®¬£2‘- _wandbruntime‚3054  _runtime‚3054I˜ÝÆ’À & lr-AdamW/pg1‚0.0007379054819365642 & lr-AdamW/pg2‚0.0007379054819365642 trainer/global_step‚3199 $ _timestamp‚1.788419583915161e+09 _runtime‚3054.665104811 _step‚258ù±>Ë“Å & lr-AdamW/pg1‚0.0007379054819365642 & lr-AdamW/pg2‚0.0007379054819365642 trainer/global_step‚3199 $ _timestamp‚1.788419583915161e+09 _runtime‚3054.665104811 _step‚258‚³7|2”- _wandbruntime‚3054  _runtime‚3054?Û¤Û'•ÕO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58594889 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125992225570816 _step‚259 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141249559440384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131922417907200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5912765835273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34276736 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83037906 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34276734 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141632797 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307442176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 _runtime‚3054.666087933 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125212946708865 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125992207567872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚652 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83037906 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55687191 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 283502080 trainer/global_step‚3199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58594891 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5929964133376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55687845 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚654 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131922417907200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119299940185088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135211809781760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131922171701248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21410457 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚125992225570816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125992207567872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261ü†wàà'–ÚO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58594889 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125992225570816 _step‚259 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141249559440384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131922417907200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5912765835273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34276736 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83037906 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34276734 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141632797 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307442176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 _runtime‚3054.666087933 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125212946708865 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125992207567872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚652 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83037906 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55687191 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 283502080 trainer/global_step‚3199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58594891 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5929964133376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55687845 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚654 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131922417907200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119299940185088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135211809781760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131922171701248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21410457 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚125992225570816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125992207567872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261ƒKþ$2—- _wandbruntime‚3055  _runtime‚3055—y5–C˜ßM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5931801827840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141650396 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 REDeviceStatsMonitor.T{“ߢ$on_train_batch_end/allocation.large_pool.allocated‚58612610 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83063483 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 _step‚260 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34287089 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275100160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚125251081566529 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58612610 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34287091 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21417804 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141676093 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126030563514880 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141650396 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6039310216192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299040256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135253003418112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884195847835374e+09 _runtime‚3055.534174303 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119336259682304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131962347339776 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚126030545511936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚3199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83063483 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83037788 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55704895 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58612608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141676093 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5932030044160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141292588734464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141292289694208 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125250840877817 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5914821884225 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126030563514880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6039585316352 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119336242642944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131962593559040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚126030545511936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135252979478016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58612608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55704241 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5914598234873 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299œ“ôGê&™äM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58612608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141676093 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5932030044160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141292588734464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141292289694208 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125250840877817 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5914821884225 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126030563514880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6039585316352 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119336242642944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131962593559040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚126030545511936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135252979478016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58612608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55704241 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5914598234873 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5931801827840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141650396 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58612610 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83063483 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 _step‚260 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34287089 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275100160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚125251081566529 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58612610 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34287091 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21417804 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141676093 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126030563514880 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141650396 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6039310216192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299040256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135253003418112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884195847835374e+09 _runtime‚3055.534174303 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119336259682304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131962347339776 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚126030545511936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚3199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83063483 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83037788 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55704895 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143401281423872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127119462484369 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 % _timestamp‚1.7884196295148277e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133930968041472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143799649 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127909122774016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127119703172873 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121115063077888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56530292 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294859264 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59480841 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6021863270400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127909104771072 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127909104771072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143401576283136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824Ó¨1—ß'¨ÙO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59480839 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34794484 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137270294594560 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143773978 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127909104771072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143401576283136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6022091473408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34794486 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6004399406481 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84293139 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84318808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133930968041472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6021863270400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270919168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84318808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84293139 _runtime‚ 3100.26519869 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59480841 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6004623055625 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21736441 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133931214247424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143401281423872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127119462484369 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 % _timestamp‚1.7884196295148277e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133930968041472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143799649 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127909122774016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127119703172873 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121115063077888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56530292 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294859264 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59480841 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6021863270400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127909104771072 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59498558 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137311488230912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137311464290816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127157597341825 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6006455455433 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6023700964864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21743114 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84344385 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6023929181184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34804841 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 _step‚264 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143842945 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84318690 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143444314530304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127157838030537 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133971143680000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143817248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6006231806081 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127947442715136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6133129533952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143444617764864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143842945 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84318690 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59498558 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56547326 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6023700964864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84344385 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 _runtime‚3101.133727946¤5樫ãM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133971389899264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6023929181184 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 trainer/global_step‚3249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59498560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133971389899264 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56547955 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6132850239488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚127947442715136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59498560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21742487 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84344385 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 _runtime‚3101.133727946 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127947460718080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59498558 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137311488230912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137311464290816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127157597341825 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6006455455433 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6023700964864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21743114 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84344385 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6023929181184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34804841 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 _step‚264 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143842945 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84318690 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 Ùcñ\èXDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143444314530304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127157838030537 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133971143680000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143817248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6006231806081 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127947442715136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6133129533952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143444617764864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143842945 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84318690 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59498558 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56547326 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6023700964864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135939764381696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6224750609408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚655 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135940010587648 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129826001974272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚122930203010048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129026459636881 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60366789 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚122930185970688 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _step‚267 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6113762407424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22056028 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85574041 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129026218948377 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6113762407424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35312236pݪtà'ºÚO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _step‚267 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6113762407424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22056028 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85574041 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129026218948377 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6113762407424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35312236 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135940010587648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139328803347456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 trainer/global_step‚3299 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135939764381696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6224750609408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚655 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135940010587648 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129826001974272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚122930203010048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129026459636881 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60366789 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚122930185970688 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0‹eŽo™2»- _wandbruntime‚3146  _runtime‚31464ʼÞM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85625287 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57385001 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35322591 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145596587707392 _step‚268 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85625287 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6097865377289 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146009797 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 307428864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _runtime‚3146.777136419 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6098089026641 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6226614663680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60384510 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129864357921280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129864357921280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6115828318208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 283488768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139369973043712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6115600101888 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6226331174912 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85599592 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145596280278528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145984100 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139369949103616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129064353805833 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146009797 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 trainer/global_step‚3299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60384508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35322589 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚122966488428544 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135979940020224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60384508 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0̻Ǧé&½ãM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145984100 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139369949103616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129064353805833 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146009797 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 trainer/global_step‚3299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60384508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35322589 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚122966488428544 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135979940020224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60384508 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85625287 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57385001 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35322591 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145596587707392 _step‚268 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85625287 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6097865377289 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146009797 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 307428864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _runtime‚3146.777136419 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6098089026641 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6226614663680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60384510 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129864357921280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129864357921280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6115828318208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 283488768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139369973043712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6115600101888 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6226331174912 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85599592 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145596280278528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561ŒùÄF2¾- _wandbruntime‚3146  _runtime‚3146º””ãZ¿Ô 0loss/FBt1/assignment_loss‚0.7088651738383553 .loss/FRt4/detection_loss‚2.309756264394643 0loss/SRqqt3/detection_loss‚2.585889968761178 /loss/FBt3/detection_loss‚5.3659080266952515 0loss/FRt4/assignment_loss‚3.5134145532335554 2loss/SRqqt2/assignment_loss‚1.1966060925403545 2loss/FBt3/assignment_loss‚0.019792674109339714 trainer/global_step‚3299 0loss/FRt2/assignment_loss‚1.8412651302767735 .loss/FRt1/detection_loss‚2.340093190710132 /loss/FRt2/detection_loss‚1.8178272453301094 .loss/FRt3/detection_loss‚3.024834742769599 1loss/SRqqt2/detection_loss‚1.2428537050968613 1loss/SRqqt4/detection_loss‚1.6684910880552757 .loss/FBt4/detection_loss‚3.166596676622118 _runtime‚3146.778143887 /loss/FBt1/detection_loss‚2.2167765579440375 0loss/FBt4/assignment_loss‚0.3086608461910925 /loss/FBt2/detection_loss‚0.5364113860453169 1loss/SRqqt3/assignment_loss‚2.066468044768932 0loss/FBt2/assignment_loss‚0.1819291928623813 epoch‚2 % _timestamp‚1.7884196760282848e+09 &loss/total_loss‚50.948047827066574 /loss/FRt3/assignment_loss‚3.428848545998335 _step‚269 2loss/SRqqt4/assignment_loss‚1.9313846978585463 1loss/SRqqt1/detection_loss‚2.6229692501947284 /loss/FRt1/assignment_loss‚4.537377148866653 2loss/SRqqt1/assignment_loss‚2.3150276239030063•ËSð_ÀÙ /loss/FBt3/detection_loss‚5.3659080266952515 0loss/FRt4/assignment_loss‚3.5134145532335554 2loss/SRqqt2/assignment_loss‚1.1966060925403545 2loss/FBt3/assignment_loss‚0.019792674109339714 trainer/global_step‚3299 0loss/FRt2/assignment_loss‚1.8412651302767735 .loss/FRt1/detection_loss‚2.340093190710132 /loss/FRt2/detection_loss‚1.8178272453301094 .loss/FRt3/detection_loss‚3.024834742769599 1loss/SRqqt2/detection_loss‚1.2428537050968613 1loss/SRqqt4/detection_loss‚1.6684910880552757 .loss/FBt4/detection_loss‚3.166596676622118 _runtime‚3146.778143887 /loss/FBt1/detection_loss‚2.2167765579440375 0loss/FBt4/assignment_loss‚0.3086608461910925 /loss/FBt2/detection_loss‚0.5364113860453169 1loss/SRqqt3/assignment_loss‚2.066468044768932 0loss/FBt2/assignment_loss‚0.1819291928623813 epoch‚2 % _timestamp‚1.7884196760282848e+09 &loss/total_loss‚50.948047827066574 /loss/FRt3/assignment_loss‚3.428848545998335 _step‚269 2loss/SRqqt4/assignment_loss‚1.9313846978585463 1loss/SRqqt1/detection_loss‚2.6229692501947284 /loss/FRt1/assignment_loss‚4.537377148866653 2loss/SRqqt1/assignment_loss‚2.3150276239030063 0loss/FBt1/assignment_loss‚0.7088651738383553 .loss/FRt4/detection_loss‚2.309756264394643 0loss/SRqqt3/detection_loss‚2.585889968761178;´|Á‚(:… ß¼äÔŒâ½x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚107.537' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5074 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Yü‚(:ô ß¼äÔ®ßë{ disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 cpu‚25.236901375465074 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 209.38671875 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125& disk.nvme0n1.out‚13275.53076171875# disk.dm-0.out‚13275.53076171875 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.59619140625 disk.dm-1.out‚ 209.34765625( proc.memory.availableMB‚ 172.17578125) disk./.usagePercent‚73.63476199737325$ disk./.usageGB‚1097.211166381836 disk.sda3.out‚ 209.34765625 disk.sda2.in‚0 disk.dm-1.in‚7.3125$ memory_percent‚98.94912242889404 proc.cpu.threads‚17 network.sent‚ 6.979752e+06 network.recv‚ 1.107013e+06 disk.sda3.in‚7.3125 disk.dm-0.in‚ 4.9296875! disk.rbd2.out‚128.44384765625 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚ 4.9296875" proc.memory.rssMB‚ 2863.2890625( disk.nvme0c0n1.out‚13275.53076171875 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625) proc.memory.percent‚17.47612953186035A4{Â(:… î¼äÔ‚è£x gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë.’ÕxÄ‚(:í î¼äÔ´šÇz disk.rbd0.out‚0 disk.sda.in‚ 7.36328125 proc.cpu.threads‚17 disk.sda3.in‚7.3125" proc.memory.rssMB‚ 2863.2890625 cpu‚25.235965542219304 disk.sda3.out‚ 209.8984375( disk./.usagePercent‚73.6347563652075 disk.rbd4.out‚0( disk.nvme0c0n1.out‚13275.90966796875 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0$ memory_percent‚98.94850254058838 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 1.59619140625 network.sent‚ 7.088869e+06# disk.dm-0.out‚13275.90966796875 disk.nvme0c0n1.in‚ 4.9296875 disk.nvme0n1.in‚ 4.9296875) proc.memory.percent‚17.47612953186035 disk.rbd3.in‚0 disk.rbd3.out‚0 network.recv‚ 1.113946e+06 disk.dm-1.out‚ 209.8984375 disk.rbd2.in‚0! disk.rbd2.out‚128.45556640625 disk.rbd4.in‚0& disk.nvme0n1.out‚13275.90966796875$ disk./.usageGB‚1097.211082458496 disk.dm-0.in‚ 4.9296875 disk.sda.out‚209.9375 disk.rbd1.in‚0( proc.memory.availableMB‚ 172.27734375¶æӱł(:… ý¼äÔ†ªýw gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.273' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4546 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07¦â¯yÆ‚(:î ý¼äÔ„¿Ë{ disk.sda3.out‚ 210.57421875 disk.rbd1.in‚0 cpu‚25.23515693920567 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd3.out‚0( disk./.usagePercent‚73.6347563652075 disk.rbd1.out‚ 1.59619140625 disk.dm-0.in‚ 4.9296875# memory_percent‚98.9490270614624 disk.rbd4.out‚0' disk.nvme0c0n1.out‚13276.2587890625 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda.out‚ 210.61328125 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 210.57421875! disk.rbd2.out‚128.58056640625 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚ 4.9296875% disk.nvme0n1.out‚13276.2587890625" proc.memory.rssMB‚ 2863.2890625 network.sent‚ 7.095178e+06 disk.rbd3.in‚0" disk.dm-0.out‚13276.2587890625 disk.sda2.in‚0( proc.memory.availableMB‚ 172.19140625) proc.memory.percent‚17.47612953186035 network.recv‚ 1.118409e+06 disk.nvme0c0n1.in‚ 4.9296875 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚1097.211082458496û„™e2Ç- _wandbruntime‚3190  _runtime‚3190h¡XÄȾ # _timestamp‚1.78841971987745e+09 _runtime‚ 3190.62735513 _step‚270 & lr-AdamW/pg1‚0.0007281507637473337 & lr-AdamW/pg2‚0.0007281507637473337 trainer/global_step‚3349Ð[RLÉÉà & lr-AdamW/pg1‚0.0007281507637473337 & lr-AdamW/pg2‚0.0007281507637473337 trainer/global_step‚3349 # _timestamp‚1.78841971987745e+09 _runtime‚ 3190.62735513 _step‚270ŽypÕ2Ê- _wandbruntime‚3190  _runtime‚3190ÌÈþ$‘ËÕO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6318265457152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130933216100889 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131742917180416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131742899177472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 284373504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86854943 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137948560721920 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131742899177472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137948560721920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚147705553617408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141387264220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚646 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3190.628291654 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚k°GJ 2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86880612 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚147705269243904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6205661544448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6205889747456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6187666548897 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86854943 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141387288160256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚124745325902848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61252739 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22376822 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6318005023744Ò¼• à'ÌÚO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚124745325902848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61252739 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22376822 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6318005023744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6318265457152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130933216100889 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131742917180416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131742899177472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 284373504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86854943 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137948560721920 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131742899177472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137948560721920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚147705553617408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141387264220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚646 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3190.628291654 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86880612 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚147705269243904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6205661544448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6205889747456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6187666548897 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86854943 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141387288160256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561Rºß2Í- _wandbruntime‚3191  _runtime‚3191T[Ì>å&ÎßM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61270460 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61270458 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148176649 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86906189 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86906189 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6189498948497 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131781237121536 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137988736360448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61270458 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚137988982579712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61270460 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6320140193792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6319869287936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148176649 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚124781628360704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22383591 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148150952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130971110269841 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148150952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6207499238912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131781255124480 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131781255124480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58223932 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131781237121536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58223279 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35840341 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141428457856512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6207499238912 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6207727455232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6207727455232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141428433916416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669ðZ5œ#ÏäM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148176649 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚124781628360704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22383591 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148150952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130971110269841 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148150952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6207499238912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131781255124480 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131781255124480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58223932 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131781237121536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58223279 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35840341 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141428457856512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6207499238912 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6207727455232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6207727455232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141428433916416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61270460 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61270458 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148176649 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86906189 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86906189 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6189498948497 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131781237121536 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137988736360448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61270458Ï2#N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚137988982579712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61270460 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6320140193792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6319869287936­’,õ2Ð- _wandbruntime‚3191  _runtime‚3191÷¤ZÑÔ 0loss/FBt1/assignment_loss‚0.6045064954087138 2loss/FBt3/assignment_loss‚0.016635289726157982 %loss/total_loss‚50.08076432373364 $ _timestamp‚1.788419720734644e+09 _runtime‚3191.484460267 .loss/FRt3/detection_loss‚2.342889635430442 .loss/FBt3/detection_loss‚4.924053470293681 /loss/FRt1/assignment_loss‚4.250473073937676 .loss/FRt4/detection_loss‚2.257256529769119 1loss/SRqqt2/detection_loss‚1.1714746611284428 _step‚273 0loss/FRt2/assignment_loss‚1.9565578902021368 1loss/FBt4/assignment_loss‚0.40723364958539604 .loss/FRt1/detection_loss‚2.385240576145324 1loss/SRqqt4/detection_loss‚1.8408183945486178 2loss/SRqqt2/assignment_loss‚1.4655971237626233 2loss/SRqqt4/assignment_loss‚2.2057201114411535 1loss/FBt2/assignment_loss‚0.18131598742444394 1loss/SRqqt3/detection_loss‚1.9665307478701815 .loss/FBt4/detection_loss‚3.005676734447479 trainer/global_step‚3349 0loss/FRt4/assignment_loss‚4.1363988336251705 2loss/SRqqt1/assignment_loss‚1.7034145509585357 2loss/SRqqt3/assignment_loss‚2.0165577974725277 /loss/FBt1/detection_loss‚2.5167293697595596 /loss/FRt3/assignment_loss‚4.063748767640855 /loss/FRt2/detection_loss‚1.6808466755963387 1loss/SRqqt1/detection_loss‚2.3722523068770385 .loss/FBt2/detection_loss‚0.608835650682026 epoch‚2*v{í_ÒÙ /loss/FRt3/assignment_loss‚4.063748767640855 /loss/FRt2/detection_loss‚1.6808466755963387 1loss/SRqqt1/detection_loss‚2.3722523068770385 .loss/FBt2/detection_loss‚0.608835650682026 epoch‚2 0loss/FBt1/assignment_loss‚0.6045064954087138 2loss/FBt3/assignment_loss‚0.016635289726157982 %loss/total_loss‚50.08076432373364 $ _timestamp‚1.788419720734644e+09 _runtime‚3191.484460267 .loss/FRt3/detection_loss‚2.342889635430442 .loss/FBt3/detection_loss‚4.924053470293681 /loss/FRt1/assignment_loss‚4.250473073937676 .loss/FRt4/detection_loss‚2.257256529769119 1loss/SRqqt2/detection_loss‚1.1714746611284428 _step‚273 0loss/FRt2/assignment_loss‚1.9565578902021368 1loss/FBt4/assignment_loss‚0.40723364958539604 .loss/FRt1/detection_loss‚2.385240576145324 1loss/SRqqt4/detection_loss‚1.8408183945486178 2loss/SRqqt2/assignment_loss‚1.4655971237626233 2loss/SRqqt4/assignment_loss‚2.2057201114411535 1loss/FBt2/assignment_loss‚0.18131598742444394 1loss/SRqqt3/detection_loss‚1.9665307478701815 .loss/FBt4/detection_loss‚3.005676734447479 trainer/global_step‚3349 0loss/FRt4/assignment_loss‚4.1363988336251705 2loss/SRqqt1/assignment_loss‚1.7034145509585357 2loss/SRqqt3/assignment_loss‚2.0165577974725277 /loss/FBt1/detection_loss‚2.5167293697595596‘HL(Ó‚(:… Œ½äÔÍÇx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.392' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o·vÔ‚(:ë Œ½äÔù«½z disk.sda3.in‚7.3125 disk.sda3.out‚ 211.140625 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625% disk./.usageGB‚1097.3391647338867 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.dm-1.out‚ 211.140625 disk.nvme0c0n1.in‚ 4.9296875) disk./.usagePercent‚73.64335207417713 disk.dm-0.in‚ 4.9296875! disk.rbd2.out‚130.10791015625 disk.rbd0.out‚0 disk.drbd1777.out‚0 network.sent‚ 7.100562e+06 disk.nvme0n1.in‚ 4.9296875) proc.memory.percent‚17.47612953186035 disk.sda.out‚ 211.1796875" disk.dm-0.out‚13378.7392578125 disk.rbd4.out‚0' disk.nvme0c0n1.out‚13378.7392578125 cpu‚25.234157404407668 proc.cpu.threads‚17 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 1.59619140625 disk.rbd0.in‚0& proc.memory.availableMB‚ 172.109375 disk.dm-1.in‚7.3125$ memory_percent‚98.94952774047852" proc.memory.rssMB‚ 2863.2890625% disk.nvme0n1.out‚13378.7392578125 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 network.recv‚ 1.121765e+06 disk.sda1.out‚0 disk.rbd3.in‚0¤ak}Õ‚(:‚ ›½äÔ˜îƒx gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.68' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ÄNÎxÖ‚(:í ›½äÔ±íÜ{ disk.nvme0c0n1.in‚ 4.9296875 disk.rbd3.out‚0 disk.sda3.out‚ 211.59375$ memory_percent‚98.94814491271973 network.sent‚ 7.209612e+06 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 211.6328125 disk.sda2.in‚0 disk.rbd1.out‚ 1.59619140625" proc.memory.rssMB‚ 2863.2890625 network.recv‚ 1.128952e+06 disk.rbd4.out‚0 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.sda2.out‚ 0.0390625) proc.memory.percent‚17.47612953186035 proc.cpu.threads‚17 disk.dm-1.in‚7.3125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚13478.47802734375 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.9296875' proc.memory.availableMB‚ 172.3359375 disk.rbd0.in‚0% disk./.usageGB‚1097.4641647338867 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd1.in‚0& disk.nvme0n1.out‚13478.47802734375# disk.dm-0.out‚13478.47802734375 disk.dm-1.out‚ 211.59375 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 4.9296875 cpu‚25.233283838833646) disk./.usagePercent‚73.65174092905913! disk.rbd2.out‚130.17041015625œ†×‚(:… ª½äÔέŽx gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.581' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5162 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°£RÕ{Ø‚(:𠪽äÔ—°Úz disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚ 4.9296875 cpu‚25.23245470171778 network.sent‚ 7.215445e+06) disk./.usagePercent‚73.65174092905913 disk.sda1.in‚ 0.05078125# disk.dm-0.out‚13478.74560546875 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚13478.74560546875# memory_percent‚98.9480972290039 disk.sda.out‚ 212.43359375 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.rbd0.out‚0) proc.memory.percent‚17.47612953186035 disk.rbd1.out‚ 1.59619140625& disk.nvme0n1.out‚13478.74560546875 disk.dm-0.in‚ 4.9296875 disk.rbd2.in‚0! disk.rbd2.out‚130.18603515625 disk.rbd4.in‚0 network.recv‚ 1.132849e+06 disk.sda3.in‚7.3125 disk.sda3.out‚ 212.39453125 disk.dm-1.out‚ 212.39453125 disk.rbd0.in‚0% proc.memory.availableMB‚ 172.34375" proc.memory.rssMB‚ 2863.2890625% disk./.usageGB‚1097.4641647338867 disk.dm-1.in‚7.3125 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 4.9296875 disk.rbd4.out‚0x ô’2Ù- _wandbruntime‚3235  _runtime‚3235OüËvÇÚÁ % _timestamp‚1.7884197649316843e+09 _runtime‚3235.681578833 _step‚274 & lr-AdamW/pg1‚0.0007247619840651453 & lr-AdamW/pg2‚0.0007247619840651453 trainer/global_step‚3399/ýiÝÌÛÆ _runtime‚3235.681578833 _step‚274 & lr-AdamW/pg1‚0.0007247619840651453 & lr-AdamW/pg2‚0.0007247619840651453 trainer/global_step‚3399 % _timestamp‚1.7884197649316843e+09’ä•_M2Ü- _wandbruntime‚3235  _runtime‚3235 SØ Û'ÝÕO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22698258 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22697600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59045334 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149857477286912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚126560431756288 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126560448795648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133659814383616 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149857174039040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59045994 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6297788884480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133659796380672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6297560681472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62138691 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88135845 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6279300120105 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88135845 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6279523769249 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 % _timestamp‚1.7884197649323816e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88161514 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62138691 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139957357062144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6297788884480 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139957603268096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143445749032960 _runtime‚3235.682514586 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139957603268096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 _step‚275 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62138689 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133659796380672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚132839972564897 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62138689 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096D<2Oà'ÞÚO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚126560431756288 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126560448795648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133659814383616 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149857174039040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59045994 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6297788884480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133659796380672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6297560681472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62138691 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88135845 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6279300120105 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88135845 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6279523769249 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 % _timestamp‚1.7884197649323816e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88161514 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62138691 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139957357062144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6297788884480 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139957603268096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143445749032960 _runtime‚3235.682514586 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139957603268096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 _step‚275 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62138689 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133659796380672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚132839972564897 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62138689 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22698258 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22697600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59045334 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149857477286912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561“àþf12ß- _wandbruntime‚3236  _runtime‚3236ÜžÆO àßM E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6281356169057 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149900205607424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150343501 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294464 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139997778919936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6299398375936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59061778 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D6DeviceStatsMonitor.on_train_batch_end/active'ù«±–.all.freed‚ 150317804 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62156410 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6299398375936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36358089 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143486942669312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 _step‚276 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59062417 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6281132519705 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚139997532700672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚3399 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139997778919936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133698152327680 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚132878107422561 _runtime‚3236.758002064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6299626592256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149900508841984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚126596751253504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚132877866733849 % _timestamp‚1.7884197660072095e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88187091 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133698134324736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143486918729216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62156408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88187091 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88161396 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6413286878208 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150343501 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62156410 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22703689 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¸Øpê&áäM MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150317804 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36358091 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6299626592256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62156408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6413566172672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88161396 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚133698134324736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚132878107422561 _runtime‚3236.758002064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6299626592256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149900508841984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚126596751253504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚132877866733849 % _timestamp‚1.7884197660072095e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88187091 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133698134324736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143486918729216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62156408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88187091 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88161396 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6413286878208 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150343501 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62156410 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22703689 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6281356169057 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149900205607424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150343501 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294464 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139997778919936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6299398375936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59061778 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150317804 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62156410 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6299398375936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36358089 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143486942669312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 _step‚276 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59062417 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6281132519705 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚139997532700672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚3399 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139997778919936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133698152327680 FÊíÄ _step‚278 % lr-AdamW/pg1‚0.000721305766076029 % lr-AdamW/pg2‚0.000721305766076029 trainer/global_step‚3449 % _timestamp‚1.7884198100158265e+09 _runtime‚3280.765750391–‹Ù×2î- _wandbruntime‚3280  _runtime‚3280_—¢K= ïÔO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6389688021504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59886673 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚135576693583872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23021187 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36865486 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152441386 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6371157340457 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚134746488340401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6389459818496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 152467057 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 285599232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152009116994048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚141966153402368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59886016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 309539328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145504257785856 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6389688021504 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89416747 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23020532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262M¶ß'ðÙO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚134746729028905 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚135576711586816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚141966153402368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59886016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 309539328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145504257785856 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6389688021504 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89416747 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23020532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6389688021504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59886673 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚135576693583872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23021187 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36865486 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528069 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152441386 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6371157340457 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚134746488340401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6389459818496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 152467057 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 285599232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152009116994048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59903323 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36875841 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135615049530880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6391297512960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89467993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135615031527936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚142006575260160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152484656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145545403542016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277197312 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135615031527936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152484656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23027482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63042360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128411857106944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128411874146304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142006329040896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63042358 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152052452188160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6391525729280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59902698 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3449 _step‚280 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89467993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6391297512960 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152510353 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6372766090913 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145545427482112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6507024706048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚134784863886569 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6372989740265 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824Ž`KUê&óäM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128411874146304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142006329040896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63042358 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152052452188160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6391525729280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59902698 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3449 _step‚280 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89467993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6391297512960 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152510353 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6372766090913 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145545427482112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6507024706048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚134784863886569 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6372989740265 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142006575260160 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚3281.624004749 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6391525729280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59903323 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36875841 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135615049530880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6391297512960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89467993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135615031527936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚142006575260160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152484656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145545403542016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277197312 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135615031527936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152484656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23027482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63042360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128411857106944˜_;.ò2ô- _wandbruntime‚3281  _runtime‚3281\tK’ õÔ 0loss/FRt4/assignment_loss‚3.4527089645465217 1loss/SRqqt2/detection_loss‚1.2660123337633336 %loss/total_loss‚49.55068878763777 epoch‚2 _runtime‚3281.625093149 % _timestamp‚1.7884198108752725e+09 2loss/SRqqt3/assignment_loss‚1.9494293634982214 1loss/FBt1/assignment_loss‚0.26960480069884885 /loss/FRt1/detection_loss‚3.2081570645858504 /loss/FRt3/detection_loss‚2.5984743237495422 /loss/FBt1/detection_loss‚2.5559960970511804 .loss/FBt3/detection_loss‚3.282367984453837 /loss/FRt3/assigûÔ:nment_loss‚3.780254046659212 2loss/SRqqt4/assignment_loss‚1.9608336091041565 /loss/FRt2/detection_loss‚1.8155624081893844 /loss/FRt4/detection_loss‚2.3910777997225523 0loss/SRqqt4/detection_loss‚1.914351775971326 .loss/FBt4/detection_loss‚3.050972616672516 trainer/global_step‚3449 1loss/SRqqt2/assignment_loss‚1.402287788545856 1loss/SRqqt3/detection_loss‚1.9890246809176777 0loss/FBt2/assignment_loss‚0.2679898018032818 1loss/FBt3/assignment_loss‚0.13966528132247427 1loss/SRqqt1/detection_loss‚2.7766782119870186 /loss/FBt2/detection_loss‚0.5247868976395537 _step‚281 /loss/FRt2/assignment_loss‚2.371701711294602 0loss/FRt1/assignment_loss‚4.2123955282671695 1loss/SRqqt1/assignment_loss‚2.096844910510949 0loss/FBt4/assignment_loss‚0.2735107866827093±Xuê_öÙ 0loss/FBt2/assignment_loss‚0.2679898018032818 1loss/FBt3/assignment_loss‚0.13966528132247427 1loss/SRqqt1/detection_loss‚2.7766782119870186 /loss/FBt2/detection_loss‚0.5247868976395537 _step‚281 /loss/FRt2/assignment_loss‚2.371701711294602 0loss/FRt1/assignment_loss‚4.2123955282671695 1loss/SRqqt1/assignment_loss‚2.096844910510949 0loss/FBt4/assignment_loss‚0.2735107866827093 0loss/FRt4/assignment_loss‚3.4527089645465217 1loss/SRqqt2/detection_loss‚1.2660123337633336 %loss/total_loss‚49.55068878763777 epoch‚2 _runtime‚3281.625093149 % _timestamp‚1.7884198108752725e+09 2loss/SRqqt3/assignment_loss‚1.9494293634982214 1loss/FBt1/assignment_loss‚0.26960480069884885 /loss/FRt1/detection_loss‚3.2081570645858504 /loss/FRt3/detection_loss‚2.5984743237495422 /loss/FBt1/detection_loss‚2.5559960970511804 .loss/FBt3/detection_loss‚3.282367984453837 /loss/FRt3/assignment_loss‚3.780254046659212 2loss/SRqqt4/assignment_loss‚1.9608336091041565 /loss/FRt2/detection_loss‚1.8155624081893844 /loss/FRt4/detection_loss‚2.3910777997225523 0loss/SRqqt4/detection_loss‚1.914351775971326 .loss/FBt4/detection_loss‚3.050972616672516 trainer/global_step‚3449 1loss/SRqqt2/assignment_loss‚1.402287788545856 1loss/SRqqt3/detection_loss‚1.9890246809176777™™ÿ£÷‚(:… æ½äÔ®ø…x gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎÄ¢mø‚(:â æ½äÔæã°z) disk./.usagePercent‚73.70220948633666 disk.rbd2.in‚0) proc.memory.percent‚17.47612953186035 disk.rbd4.in‚0 disk.rbd4.out‚0$ disk.nvme0c0n1.out‚ 13776.8828125 disk.sda3.out‚ 214.7890625 disk.rbd0.in‚0% disk./.usageGB‚1098.2161827087402 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0" disk.nvme0n1.out‚ 13776.8828125 disk.dm-1.in‚7.3125 disk.drbd1777.out‚0' proc.memory.availableMB‚ 172.2734375 proc.cpu.threads‚17 network.recv‚ 1.152249e+06 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.rbd1.out‚ 1.59619140625 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.sda.out‚ 214.828125 disk.dm-0.out‚ 13776.8828125 disk.dm-1.out‚ 214.7890625 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd2.out‚131.9443359375$ memory_percent‚98.94852638244629" proc.memory.rssMB‚ 2863.2890625 cpu‚25.22907846311016 disk.sda.in‚ 7.36328125 disk.sda2.in‚0 network.sent‚ 7.341979e+06 disk.nvme0n1.in‚ 4.9296875 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625¥qQù‚(:… õ½äÔÛÀx gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U.]|ú‚(:ñ õ½äÔŶ¶z disk.rbd4.in‚0 disk.sda.in‚ 7.36328125# disk.dm-0.out‚13876.08447265625 disk.rbd4.out‚0 disk.rbd0.in‚0$ memory_percent‚98.94881248474121) proc.memory.percent‚17.47612953186035( disk.nvme0c0n1.out‚13876.08447265625 disk.sda3.in‚7.3125 disk.rbd3.out‚0 disk.sda.out‚ 215.50390625 disk.rbd1.out‚ 1.59619140625 disk.nvme0n1.in‚ 4.9296875 disk.sda2.in‚0 disk.dm-1.out‚ 215.46484375 disk.drbd1777.in‚ 1.03515625 disk.rbd1.in‚0& disk.nvme0n1.out‚13876.08447265625 disk.dm-0.in‚ 4.9296875 disk.rbd2.in‚0" proc.memory.rssMB‚ 2863.2890625 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 proc.cpu.threads‚17 network.sent‚ 7.451352e+06) disk./.usagePercent‚73.70207533838871$ disk./.usageGB‚1098.214183807373 disk.sda1.out‚0 disk.dm-1.in‚7.3125 disk.rbd2.out‚132.0771484375 disk.rbd0.out‚0 cpu‚25.22819503500408 disk.sda3.out‚ 215.46484375 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0' proc.memory.availableMB‚ 172.2265625 network.recv‚ 1.159111e+06Ô¿Gû‚(:… „¾äÔƱx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?a0Œ~ü‚(:ó „¾äÔŠÔ‡{ disk.sda2.in‚0 disk.dm-1.out‚ 216.12890625 disk.rbd2.in‚0 network.sent‚ 7.456618e+06) disk./.usagePercent‚73.70208148256954 disk.rbd3.in‚0 disk.sda2.out‚ 0.0390625( disk.nvme0c0n1.out‚13975.23291015625$ memory_percent‚98.95033836364746% disk./.usageGB‚1098.2142753601074 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚ 4.9296875' proc.memory.availableMB‚ 171.9765625 disk.sda3.in‚7.3125 disk.sda3.out‚ 216.12890625 disk.rbd1.out‚ 1.59619140625 disk.dm-0.in‚ 4.9296875 disk.dm-1.in‚7.3125 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd3.out‚0 disk.sda.out‚ 216.16796875 disk.rbd2.out‚132.0927734375 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0" proc.memory.rssMB‚ 2863.2890625) proc.memory.percent‚17.47612953186035 cpu‚25.227385061485986 network.recv‚ 1.162768e+06 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚13975.23291015625# disk.dm-0.out‚13975.23291015625 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.in‚0d¸"¸2ý- _wandbruntime‚3325  _runtime‚3325mQŽªÇþÁ trainer/global_step‚3499 % _timestamp‚1.7884198551518373e+09 _runtime‚3325.901739524 _step‚282 & lr-AdamW/pg1‚0.0007177828300506336 & lr-AdamW/pg2‚0.0007177828300506336 ±_ÌÿÆ & lr-AdamW/pg1‚0.0007177828300506336 & lr-AdamW/pg2‚0.0007177828300506336 trainer/global_step‚3499 % _timestamp‚1.7884198551518373e+09 _runtime‚3325.901739524 _step‚282šÓ¦ 2€- _wandbruntime‚3325  _runtime‚33254G?oÚ'ÔO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154161419178496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273016320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23343779 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884198551525445e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63910591 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137493608790016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚143974949742592 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130190694581248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 trainer/global_step‚3499 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63910591 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63910589 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143975195948544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143975195948544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147562742598656 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 3325.90262398 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154633909 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63910589 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60727698 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37383234 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6598403563520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6481587158528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60727013 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143974949742592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137493608790016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6598676579840 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚283 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130190677541888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6462567262521 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296956416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669Ë‚èîß'‚ÙO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63910589 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60727698 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37383234 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6598403563520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6481587158528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60727013 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143974949742592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137493608790016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6598676579840 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚283 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130190677541888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6462567262521 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296956416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154161419178496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273016320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23343779 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884198551525445e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63910591 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137493608790016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚143974949742592 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130190694581248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 trainer/global_step‚3499 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63910591 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63910589 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143975195948544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143975195948544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147562742598656 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 3325.90262398 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154633909 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137531946734080 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154651508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63928310 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 % _timestamp‚1.7884198560130844e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144015371600384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154204469870080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6464399662121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90748895 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37393591 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6464623311473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144015371600384 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6483196649984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6483424866304 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90723200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154677205 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147603912294912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6483196649984 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60744384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23350150 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6483424866304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60743739 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63928308 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63928308 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147603888354816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277197312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154677205 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63928310 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 trainer/global_step‚3499 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137531928731136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6600557575168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137531946734080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 _runtime‚3326.763732461 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 trainer/global_step‚3499 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137531928731136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6600557575168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137531946734080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 _runtime‚3326.763732461 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137531946734080 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154651508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63928310 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 % _timestamp‚1.7884198560130844e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144015371600384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154204469870080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6464399662121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90748895 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37393591 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6464623311473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144015371600384 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6483196649984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6483424866304 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90723200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154677205 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147603912294912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6483196649984 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60744384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23350150 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6483424866304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60743739 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63928308 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63928308 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147603888354816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277197312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154677205 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63928310 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64796539 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6692160528896 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139410487990272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145983746082816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64796541 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132005NêéE800434688 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156313086789632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139410505993216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚3370.991592311D½Áà'”ÚO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6554424482873 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294669 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139410487990272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64796541 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91978551 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156775090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145983746082816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64796541 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132005800434688 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25669 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156313086789632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139410505993216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚3370.991592311 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92004220 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6554200833729 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚138560241956921 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚145983992288768 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31396 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6573258092544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37900984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61571299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23670313 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 _step‚287 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23669652 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6573486295552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139410505993216 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31396 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132005817474048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64796539 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6692160528896 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139410487990272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132042102892544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6556256882681 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92029797 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139448843937280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156356420593152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 _runtime‚3371.854726371 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92004102 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132042119931904 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149662397107712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149662373167616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64814258 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6575095787008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146024167940608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156844057 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92004102 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156818360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64814260 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6556033233329 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚139448825934336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61588033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37911341 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37911339 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294464 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 $ _timestamp‚1.788419901104318e+09 _step‚288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156356117358592 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146024167940608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139448843937280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61587365 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6694023485440 trainer/global_step‚3549 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156844057 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23676026 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚138598376814585 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ÖCªÝé&—ãM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156844057 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92004102 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25697 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156818360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64814260 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6556033233329 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚139448825934336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61588033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37911341 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37911339 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279294464 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25695 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 $ _timestamp‚1.788419901104318e+09 _step‚288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31396 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156356117358592 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146024167940608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25695 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139448843937280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61587365 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6694023485440 trainer/global_step‚3549 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156844057 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23676026 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚138598376814585 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132042102892544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6556256882681 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92029797 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139448843937280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156356420593152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 _runtime‚3371.854726371 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92004102 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132042119931904 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149662397107712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149662373167616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64814258 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6575095787008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146024167940608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649352 @>*‹2˜- _wandbruntime‚3371  _runtime‚3371ÖwO­Z™Ô % _timestamp‚1.7884199011061702e+09 -loss/FRt1/detection_loss‚2.76970733212013 0loss/FRt3/assignment_loss‚3.2707991527788565 2loss/SRqqt3/assignment_loss‚1.7522641329980286 1loss/SRqqt2/detection_loss‚1.0380736119943121 1loss/SRqqt4/detection_loss‚1.9516127055341548 /loss/FBt2/detection_loss‚0.7352477874964067 0loss/SRqqt3/detection_loss‚1.758730509730636 trainer/global_step‚3549 0loss/FRt2/assignment_loss‚2.0277751175065837 1loss/SRqqt1/assignment_loss‚2.320394427277321 2loss/SRqqt4/assignment_loss‚1.7071254707195542 .loss/FBt1/detection_loss‚2.307478118687868 epoch‚2 _runtime‚3371.856037601 0loss/FRt1/assignment_loss‚3.6964589169150903 2loss/SRqqt2/assignment_loss‚1.1793083803830287 _step‚289 1loss/FBt2/assignment_loss‚0.20760547921378927 0loss/FBt3/assignment_loss‚0.0262770460394659 /loss/FRt2/detection_loss‚1.7913715629838407 /loss/FRt4/assignment_loss‚4.221185710923425 /loss/FRt3/detection_loss‚2.6603949837612384 .loss/FRt4/detection_loss‚2.937906300199443 1loss/SRqqt1/detection_loss‚2.1793872589288754 .loss/FBt3/detection_loss‚3.459103971719742 /loss/FBt4/detection_loss‚2.3090999245643617 0loss/FBt1/assignment_loss‚0.7025227670092136 1loss/FBt4/assignment_loss‚0.45071365920284734 &loss/total_loss‚47.460544328688215ÓçÎNšÙ 1loss/FBt2/assignment_loss‚0.20760547921378927 0loss/FBt3/assignment_loss‚0.0262770460394659 /loss/FRt2/detection_loss‚1.7913715629838407 /loss/FRt4/assignment_loss‚4.221185710923425 /loss/FRt3/detection_loss‚2.6603949837612384 .loss/FRt4/detection_loss‚2.937906300199443 1loss/SRqqt1/detection_loss‚2.1793872589288754 .loss/FBt3/detection_loss‚3.459103971719742 /loss/FBt4/detection_loss‚2.3090999245643617 0loss/FBt1/assignment_loss‚0.7025227670092136 1loss/FBt4/assignment_loss‚0.45071365920284734 &loss/total_loss‚47.46054432868821°V{»5 % _timestamp‚1.7884199011061702e+09 -loss/FRt1/detection_loss‚2.76970733212013 0loss/FRt3/assignment_loss‚3.2707991527788565 2loss/SRqqt3/assignment_loss‚1.7522641329980286 1loss/SRqqt2/detection_loss‚1.0380736119943121 1loss/SRqqt4/detection_loss‚1.9516127055341548 /loss/FBt2/detection_loss‚0.7352477874964067 0loss/SRqqt3/detection_loss‚1.758730509730636 trainer/global_step‚3549 0loss/FRt2/assignment_loss‚2.0277751175065837 1loss/SRqqt1/assignment_loss‚2.320394427277321 2loss/SRqqt4/assignment_loss‚1.7071254707195542 .loss/FBt1/detection_loss‚2.307478118687868 epoch‚2 _runtime‚3371.856037601 0loss/FRt1/assignment_loss‚3.6964589169150903 2loss/SRqqt2/assignment_loss‚1.1793083803830287 _step‚289¡Ø)ëQ›‚(:… À¾äÔ¡ó™x gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÖ€yœ‚(:î À¾äÔ’ôçz# disk.dm-0.out‚14174.85107421875 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚73.70210093914213& disk.nvme0n1.out‚14174.85107421875 disk.sda2.in‚0 disk.dm-1.in‚7.3125) proc.memory.percent‚17.47615337371826 cpu‚25.2241129501527 network.recv‚ 1.182242e+06# memory_percent‚98.9516019821167( proc.memory.availableMB‚ 171.76953125 disk.sda3.out‚ 218.609375 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 7.583187e+06 disk.rbd1.in‚0 disk.dm-0.in‚ 4.9296875 disk.dm-1.out‚ 218.609375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚14174.85107421875 disk.sda3.in‚7.3125! disk.rbd2.out‚133.86669921875 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0# proc.memory.rssMB‚ 2863.29296875% disk./.usageGB‚1098.2145652770996 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 1.59619140625 disk.nvme0n1.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.rbd2.in‚0 disk.sda.out‚ 218.6484375/Îè4š‚(: ϾäÔ÷°€x gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.465' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.493000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>IK¬wž‚(:ì ϾäÔ¬äz disk.rbd4.in‚0# disk.dm-0.out‚14274.44189453125 disk.sda2.in‚0 disk.drbd1777.out‚0) proc.memory.percent‚17.47615337371826 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625( proc.memory.availableMB‚ 171.87890625 proc.cpu.threads‚17 disk.sda.out‚ 219.0703125 disk.sda3.in‚7.3125 disk.rbd1.out‚ 1.59619140625) disk./.usagePercent‚73.70207354633597& disk.nvme0n1.out‚14274.44189453125 disk.sda1.out‚0# proc.memory.rssMB‚ 2863.29296875 network.sent‚ 7.692049e+06 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 219.03125 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.nvme0c0n1.in‚ 4.9296875 cpu‚25.223275835863973 disk.rbd4.out‚0 network.recv‚ 1.188735e+06 disk.rbd3.out‚0 disk.dm-1.out‚ 219.03125 disk.rbd2.in‚0( disk.nvme0c0n1.out‚14274.44189453125% disk./.usageGB‚1098.2141571044922 disk.dm-0.in‚ 4.9296875 disk.dm-1.in‚7.3125 disk.rbd2.out‚ 133.95703125 disk.sda1.in‚ 0.05078125$ memory_percent‚98.95093441009521 disk.rbd3.in‚0È}¸Ÿ‚(:… Þ¾äÔäðàx gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.896' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óéøy ‚(:î Þ¾äÔݶ»z disk.sda3.in‚7.3125$ memory_percent‚98.95250797271729) proc.memory.percent‚17.47615337371826 disk.rbd2.out‚ 133.95703125 disk.drbd1777.out‚0 disk.sda3.out‚ 219.640625& disk.nvme0n1.out‚14373.53955078125 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 1.59619140625) disk./.usagePercent‚73.70207354633597 disk.rbd2.in‚0( disk.nvme0c0n1.out‚14373.53955078125 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 219.6796875 disk.dm-0.in‚ 4.9296875 disk.rbd1.in‚0 network.sent‚ 7.697944e+06% disk./.usageGB‚1098.2141571044922 disk.nvme0n1.in‚ 4.9296875# disk.dm-0.out‚14373.53955078125 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9296875 cpu‚25.222523261259298 disk.rbd0.out‚0# proc.memory.rssMB‚ 2863.29296875 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.out‚0( proc.memory.availableMB‚ 171.62109375 proc.cpu.threads‚17 network.recv‚ 1.192956e+06 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 219.640625 disk.rbd0.in‚0 disk.rbd3.in‚0>§õh2¡- _wandbruntime‚3416  _runtime‚3416ð™ßÄ¢¾ $ _timestamp‚1.788419945483138e+09 _runtime‚ 3416.2330752 _step‚290 & lr-AdamW/pg1‚0.0007105397543402928 & lr-AdamW/pg2‚0.0007105397543402928 trainer/global_step‚3599MY6ɣà & lr-AdamW/pg2‚0.0007105397543402928 trainer/global_step‚3599 $ _timestamp‚1.788419945483138e+09 _runtime‚ 3416.2330752 _step‚290 & lr-AdamW/pg1‚0.0007105397543402928¢¢>^·2¤- _wandbruntime‚3416  _runtime‚3416y2Ò|Û'¥ÕO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6785629611008 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275113472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62410069 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6645834404937 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚652 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65682491 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38418734 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141327385193472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚3599 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6665385432576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6785354497536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚3416.234066878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚141327403196416 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6665385432576 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚133820923327488 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _step‚291 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147992542423040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65682491 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚141327403196416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23990681 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93285122 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93259453 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053568 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158941942 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141327385193472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158465042781696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚147992788628992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚147992542423040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65682489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23991333 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0HÝ1à'¦ÚO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚141327403196416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23990681 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93285122 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93259453 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053568 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649144 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158941942 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141327385193472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158465042781696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚147992788628992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚147992542423040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65682489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23991333 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6785629611008 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275113472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62410069 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25671 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6645834404937 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246205952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚652 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65682491 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38418734 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25671 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141327385193472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31167 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚3599 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344384 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246205952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6665385432576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6785354497536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚3416.234066878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚141327403196416 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31167 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6665385432576 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚133820923327488 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _step‚291 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147992542423040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25669 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65682491 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0£B§ï2§- _wandbruntime‚3417  _runtime‚3417v»‡…¨ßM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294669 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65700208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158985212 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148032964280832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 trainer/global_step‚3599 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6667223140352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159010909 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133857225785344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6787496524800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6647666804537 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93310699 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚141365741140480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141365723137536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159010909 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65700210 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62426138 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158508068919296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151720881920512 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚140504892589881 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148032718061568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 _runtime‚3417.096872654 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93285004 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62426813 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884199463460941e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6666994924032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 _step‚292 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141365723137536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚140505133278593 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38429091 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285585920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133857242824704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6666994924032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148032964280832 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65700210 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159010909 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65700210 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62426138 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158508068919296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151720881920512 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚140504892589881 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344384 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148032718061568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216320 _runtime‚3417.096872654 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93285004 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62426813 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884199463460941e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6666994924032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 _step‚292 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31396 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141365723137536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31167 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚140505133278593 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38429091 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285585920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133857242824704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6666994924032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148032964280832 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344384 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65700210 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528069 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159010909 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219264¤„¶ë‘2ª- _wandbruntime‚3417  _runtime‚3417/ªÃëU«Ï /loss/FRt2/detection_loss‚2.3611917005230985 1loss/SRqqt3/detection_loss‚1.9294286009210806 0loss/SRqqt4/detection_loss‚1.870795047879219 _runtime‚3417.097913285 0loss/FRt3/assignment_loss‚3.8420403487980366 /loss/FBt4/detection_loss‚2.1420212388038635 _step‚293 /loss/FRt4/assignment_loss‚3.519754900932312 2loss/FBt3/assignment_loss‚0.021919799968600273 0loss/SRqqt1/detection_loss‚2.643741875886917 .loss/FRt1/detection_loss‚2.502218242280367 ,loss/FRt3/detection_loss‚2.8638965934515 /loss/FRt4/detection_loss‚2.1547652530670165 /loss/FBt2/detection_loss‚0.7112011466683312 %loss/total_loss‚48.44393681500864 1loss/FBt2/assignment_loss‚0.11828816095272703 epoch‚2 2loss/SRqqt2/assignment_loss‚1.3092542813453627 1loss/SRqqt3/assignment_loss‚2.318966654630808 /loss/FBt3/detection_loss‚3.4442796409130096 1loss/SRqqt2/detection_loss‚0.9303039561368918 -loss/FBt1/detection_loss‚2.36199223001798 /loss/FRt1/assignment_loss‚4.052101141068993 1loss/SRqqt1/assignment_loss‚1.976547730083649 2loss/SRqqt4/assignment_loss‚1.6299583914875984 trainer/global_step‚3599 % _timestamp‚1.7884199463480666e+09 /loss/FRt2/assignment_loss‚2.252250807152854 0loss/FBt1/assignment_loss‚0.8277385880549749 0loss/FBt4/assignment_loss‚0.6592804839834571Å™½Z¬Ô 2loss/SRqqt2/assignment_loss‚1.3092542813453627 1loss/SRqqt3/assignment_loss‚2.318966654630808 /loss/FBt3/detection_loss‚3.4442796409130096 1loss/SRqqt2/detection_loss‚0.9303039561368918 -loss/FBt1/detection_loss‚2.36199223001798 /loss/FRt1/assignment_loss‚4.052101141068993 1loss/SRqqt1/assignment_loss‚1.976547730083649 2loss/SRqqt4/assignment_loss‚1.6299583914875984 trainer/global_step‚3599 % _timestamp‚1.7884199463480666e+09 /loss/FRt2/assignment_loss‚2.252250807152854 0loss/FBt1/assignment_loss‚0.8277385880549749 0loss/FBt4/assignment_loss‚0.6592804839834571 /loss/FRt2/detection_loss‚2.3611917005230985 1loss/SRqqt3/detection_loss‚1.9294286009210806 0loss/SRqqt4/detection_loss‚1.870795047879219 _runtime‚3417.097913285 0loss/FRt3/assignment_loss‚3.8420403487980366 /loss/FBt4/detection_loss‚2.1420212388038635 _step‚293 /loss/FRt4/assignment_loss‚3.519754900932312 2loss/FBt3/assignment_loss‚0.021919799968600273 0loss/SRqqt1/detection_loss‚2.643741875886917 .loss/FRt1/detection_loss‚2.502218242280367 ,loss/FRt3/detection_loss‚2.8638965934515 /loss/FRt4/detection_loss‚2.1547652530670165 /loss/FBt2/detection_loss‚0.7112011466683312 %loss/total_loss‚48.44393681500864 1loss/FBt2/assignment_loss‚0.11828816095272703 epoch‚2¥‹È¹Q­‚(:… í¾äÔé¿x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.792' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ*$‹x®‚(:í í¾äÔÐÒÂ{ disk.rbd1.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625# proc.memory.rssMB‚ 2774.08203125% disk./.usageGB‚1098.1790657043457 disk.sda.in‚ 7.36328125# disk.dm-0.out‚14373.83837890625 disk.dm-1.in‚7.3125 network.sent‚ 7.704298e+06) disk./.usagePercent‚73.69971853302849 disk.sda3.in‚7.3125 disk.sda3.out‚ 220.03125& disk.nvme0n1.out‚14373.83837890625 disk.dm-1.out‚ 220.03125 disk.rbd2.out‚ 134.03515625( proc.memory.availableMB‚ 260.55859375 network.recv‚ 1.197507e+06 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚ 4.9296875( disk.nvme0c0n1.out‚14373.83837890625 disk.rbd3.out‚0 disk.rbd1.out‚ 1.59619140625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 220.0703125 disk.nvme0n1.in‚ 4.9296875 disk.rbd0.in‚0* proc.memory.percent‚16.931653022766113$ memory_percent‚98.40967655181885 proc.cpu.threads‚17 disk.rbd2.in‚0 cpu‚25.221828814911078 disk.sda2.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0­³>¯‚(:… ü¾äÔýÚ­x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06ÆŠZw°‚(:ì ü¾äÔÈ…°z$ disk./.usageGB‚1098.179401397705% disk.nvme0n1.out‚14473.1533203125 disk.dm-1.in‚7.3125 disk.rbd4.out‚0 network.sent‚ 7.812479e+06 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda3.in‚7.3125# proc.memory.rssMB‚ 2774.08203125 disk.rbd1.out‚ 1.59619140625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda.in‚ 7.36328125 disk.sda.out‚ 220.76171875 disk.sda3.out‚ 220.72265625* proc.memory.percent‚16.931653022766113 cpu‚25.221029351268005 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 4.9296875" disk.dm-0.out‚14473.1533203125 disk.dm-1.out‚ 220.72265625 disk.rbd3.in‚0' proc.memory.availableMB‚ 261.1484375 disk.rbd0.out‚0' disk.nvme0c0n1.out‚14473.1533203125 disk.sda1.out‚0 network.recv‚ 1.203308e+06( disk./.usagePercent‚73.6997410616915 disk.nvme0n1.in‚ 4.9296875$ memory_percent‚98.40607643127441 disk.rbd2.out‚ 135.546875 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625ŽN_¸±‚(:„ ‹¿äԾϦx gpu.0.gpu‚46.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.498' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e_mt²‚(:é ‹¿äÔ©‚Žz disk.rbd3.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0' proc.memory.availableMB‚ 162.8984375 disk.drbd1777.in‚ 1.03515625 disk.dm-1.out‚ 221.1640625 network.sent‚ 7.818537e+06 disk.sda.in‚ 7.36328125 disk.sda.out‚ 221.203125 disk.nvme0n1.in‚ 4.9296875 disk.rbd4.in‚0 disk.sda3.out‚ 221.1640625 disk.dm-0.in‚ 4.9296875 proc.cpu.threads‚17 disk.rbd0.in‚0$ memory_percent‚99.00574684143066) disk./.usagePercent‚73.69974157370658 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda2.in‚0 disk.dm-1.in‚7.3125 cpu‚25.232089050432055# disk.dm-0.out‚14572.94677734375 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda3.in‚7.3125& disk.nvme0n1.out‚14572.94677734375 disk.rbd2.in‚0! proc.memory.rssMB‚ 2872.140625* proc.memory.percent‚17.530155181884766 network.recv‚ 1.208192e+06% disk./.usageGB‚1098.1794090270996 disk.rbd4.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.59619140625 disk.rbd2.out‚135.6875 disk.nvme0c0n1.in‚ 4.9296875( disk.nvme0c0n1.out‚14572.94677734375.Ä厳‚(:ƒ š¿äÔžñ¦x gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.424' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àªaq´‚(:æ š¿äÔô‘Ô{ network.sent‚ 7.823509e+06 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 183.93359375) proc.memory.percent‚17.39976406097412 network.recv‚ 1.21152e+06& disk.nvme0n1.out‚14672.18310546875 disk.sda2.in‚0 disk.rbd2.out‚135.6875 disk.sda3.in‚7.3125 disk.sda3.out‚ 221.84375# proc.memory.rssMB‚ 2850.77734375) disk./.usagePercent‚73.69973645355589 disk.sda1.in‚ 0.05078125( disk.nvme0c0n1.out‚14672.18310546875 disk.sda1.out‚0 disk.sda.in‚ 7.36328125$ memory_percent‚98.87735843658447 cpu‚25.25054485979375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4.9296875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.9296875% disk./.usageGB‚1098.1793327331543 disk.rbd1.out‚ 1.59619140625# disk.dm-0.out‚14672.18310546875 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 221.84375 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.dm-0.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 221.8828125„kˆËŽµ‚(:ƒ ©¿äÔçÕ±x gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.424' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡c1 m¶‚(:â ©¿äÔâéŒ| disk.rbd1.out‚ 1.59619140625 disk.nvme0n1.in‚ 4.9296875 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 4.9296875# proc.memory.rssMB‚ 2850.77734375 cpu‚25.267647031484575 proc.cpu.threads‚17% disk./.usageGB‚1098.1774520874023$ disk.nvme0c0n1.out‚ 14672.8203125 disk.sda.in‚ 7.36328125) proc.memory.percent‚17.39976406097412 disk.sda.out‚222.3125 disk.sda3.in‚7.3125 network.recv‚ 1.21633e+06 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 222.2734375 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.dm-0.out‚ 14672.8203125 disk.rbd0.out‚0 disk.sda2.in‚0! disk.rbd2.out‚135.85595703125' proc.memory.availableMB‚ 183.9453125 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0" disk.nvme0n1.out‚ 14672.8203125) disk./.usagePercent‚73.69961024184151 disk.sda2.out‚ 0.0390625 disk.rbd4.out‚0$ memory_percent‚98.87728691101074 network.sent‚ 7.829848e+06 disk.sda3.out‚ 222.2734375 disk.rbd1.in‚0E2³Ó2·- _wandbruntime‚3493  _runtime‚34938-®¨ !¸¼T UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚94406547 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20200 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 541892096 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20200 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚136050286660608 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚150417682250752 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚94406547 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1243 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 167821396 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚299 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚66910485 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6736107635712 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚161033016099328 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚94386347 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 151250944 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 537294669 _runtime‚3493.594255677 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚39147910 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚161033191290368 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 6893338624 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚63528856 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚142766672653193 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6522528069 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20202 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 171710464 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 161317034 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3428 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31167 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 161296832 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚154175798321152 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 161317034 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚150417871964160 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚66910487 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 184860756 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 189713408 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 171710464 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚381 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6739344384 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚262 % _timestamp‚1.7884200228437448e+09 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚561 JDDeviceStatsMonitor.on_va°ˆõ" lidation_batch_start/oversize_segments.freed‚0 trainer/global_step‚3626 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚150417682250752 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 6893338624 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚143681764328448 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20202 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 161296832 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚561 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚299 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6735935925248 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚63529239 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚23 ^QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚90898432 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚136050303699968 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚561 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6739344384 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚66910487 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 549453824 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 189713408 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚143681746325504 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚24381327 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚299³ÄG*¹ÁT ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚150417871964160 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 6857241718272 _step‚294 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚561 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31396 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1261 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 6735935925248 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚66910485 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚24380946 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 6893338624 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚143681764328448 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚39147912 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 175191040 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚154175774381056 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚94386347 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6736107635712 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6343884800 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31167 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 6857392969216 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 6716385992585 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6252512256 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚383 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚142766857513949 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚299 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚66910485 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6736107635712 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚161033016099328 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚94386347 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 151250944 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 537294669 _runtime‚3493.594255677 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚39147910 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚161033191290368 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 6893338624 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚63528856 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚142766672653193 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6522528069 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20202 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 171710464 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 161317034 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3428 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31167 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 161296832 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚154175798321152 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 161317034 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚150417871964160 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚66910487 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 184860756 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 189713408 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 171710464 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚381 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6739344384 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚262 % _timestamp‚1.7884200228437448e+09 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚561 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 trainer/global_step‚3626 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚150417682250752 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 6893338624 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚143681764328448 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20202 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 161296832 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚561 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚299 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6735935925248 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚63529239 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚23 ^QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚90898432 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚136050303699968 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚561 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6739344384 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚66910487 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 549453824 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 189713408 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚143681746325504 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚24381327 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚299 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚94406547 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20200 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 541892096 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20200 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚136050286660608 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚150417682250752 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚94406547 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1243 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 167821396 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31396 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚94394623 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚142791816779337 _step‚295 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚384 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 6857785764352 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚136075056649216 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚39156876 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337968128 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚94394623 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 189713408 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚66925423 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚66925421 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 6736500430848 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6739344384 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20202 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 537294669 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚150444215835648 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 161320044 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚66925423 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 6857634513408 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20202 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚66925421 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚05ñ‚ö"¼ËR @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚94394623 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 189713408 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚66925423 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚66925421 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 6736500430848 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6739344384 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20202 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 537294669 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚150444215835648 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 161320044 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚66925423 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 6857634513408 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20202 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚66925421 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚150444215835648 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚143707905118208 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 161320044 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚24385233 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 6716944990877 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚143707905118208 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 161340246 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 151250944 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚561 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31396 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚94394623 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚142791816779337 _step‚295 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚384 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 6857785764352 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚136075056649216 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚39156876 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337968128§mMjŽ½‚(:ƒ ¸¿äÔݽ x gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.424' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gš7l¾‚(:á ¸¿äÔ»¤±{ disk.rbd4.in‚0$ memory_percent‚98.87728691101074 disk.dm-0.in‚ 4.9296875! disk.dm-0.out‚14772.208984375 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.sda3.out‚222.9375$ disk./.usageGB‚1098.177562713623 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.drbd1777.in‚ 1.03515625' proc.memory.availableMB‚ 183.9453125 network.sent‚ 7.835159e+06 disk.dm-1.out‚222.9375 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875& disk.nvme0c0n1.out‚14772.208984375 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9296875 network.recv‚ 1.219593e+06& disk./.usagePercent‚73.69961766606 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda.out‚ 222.9765625# proc.memory.rssMB‚ 2850.77734375 disk.rbd3.in‚0 disk.rbd4.out‚0) proc.memory.percent‚17.39976406097412 proc.cpu.threads‚17 disk.rbd1.out‚ 1.59619140625 cpu‚25.28540190173906 disk.rbd0.out‚0$ disk.nvme0n1.out‚14772.208984375! disk.rbd2.out‚135.85595703125ý"ôª2¿- _wandbruntime‚3508  _runtime‚3508‡ðÓ®VsÀ ¥ ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3709937632083893 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3586309552192688 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 5validation_average_jet_accuracy‚0.341006338596344 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3991319239139557 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5075757503509521 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0317460335791111 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 -jet/accuracy_4_of_4‚0.0008818342466838658 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636362552642822 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.987565815448761 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15611879527568817 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3035714328289032 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02467358112335205 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.21604938805103302 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2777256667613983 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.37176090478897095 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9897756576538086 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9900497198104858 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3074030876159668 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0297619067132473 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.810495913028717 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2361111044883728 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.008543875999748707 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 )jet/accuracy_1_of_5‚0.879621148109436 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07999999821186066 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0968480259180069 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.992092490196228 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.30778229236602783 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10905350744724274 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_6‚0.9294872283935547 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.006593769416213036 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05480670556426048 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.25585582852363586 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17676766216754913 particle/accuracy_1_of_5‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31827235221862793 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_5_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2923413813114166 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.03270745277404785 particle/accuracy_2_of_3‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.2903914451599121 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2777777910232544 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11198775470256805 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5075757503509521 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12829822301864624 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.09130623191595078 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10818713158369064 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.31609195470809937 particle/accuracy_1_of_4‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.170929417014122 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9876256585121155 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2687363624572754 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9883720278739929 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.009803921915590763 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9733333587646484 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.40468207001686096 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2578543722629547 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11786902695894241 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9914483428001404 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.42596161365509033 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 )loss/val_total_loss‚48.62661683198177 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.17968539893627167 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.30106136202812195 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.028368795290589333 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9903141260147095 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.023418273776769638 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11453457921743393 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13494668900966644 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.33666664361953735 jet/accuracy_6_of_7‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3266666829586029 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0022338533308357 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_6‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16996479034423828 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.2948477268218994 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.991067111492157 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2482256144285202 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12573380768299103 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.2903914451599121 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.277045339345932 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.01360864844173193 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4208333492279053 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.381944477558136 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.01810620166361332 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 particle/accuracy_3_of_7‚1 particle/accuracy_7_of_7‚1 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.26068660616874695 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.33624032139778137 )jet/accuracy_1_of_2‚0.607465922832489 jet/accuracy_7_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9785714745521545 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 jet/accuracy_5_of_6‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.30000001192092896 jet/accuracy_2_of_7‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 *jet/accuracy_2_of_5‚0.3837749660015106 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9460555911064148 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3769063651561737 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.29973456263542175 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008818342466838658 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1845327615737915 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4769168794155121 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.268627405166626 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.36170637607574463 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1644144058227539 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7327113747596741 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4208090007305145 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1018165871500969 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12729884684085846 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.0768401250243187 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4120071530342102 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9954621195793152 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9845237135887146 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.27470239996910095 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09433633834123611 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.38864102959632874 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 <"Purity/0FRt2SRqqt1FBt/event_purity‚0.0026954179629683495 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2588961720466614 particle/accuracy_5_of_6‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2645302712917328 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 particle/accuracy_3_of_3‚1 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 +jet/accuracy_2_of_4‚0.23339654505252838 % _timestamp‚1.7884200377192292e+09 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09813167154788971 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2742424011230469 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_7‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.005708481650799513 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10181235522031784 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04412972182035446 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09912828356027603 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2803189754486084 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2770157754421234 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15914630889892578 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11058702319860458 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6217588186264038 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.006236972752958536 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.27619048953056335 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2657490074634552 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3343355059623718 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.47467362880706787 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04046492651104927 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10859786719083786 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3564814627170563 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2629435360431671 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.259869784116745 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.32466843724250793 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2423035055398941 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11385779082775116 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22636811435222626 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.255495548248291 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.26747533679008484 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2621200382709503 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 particle/accuracy_2_of_7‚1 *"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9885181188583374 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.38871318101882935 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9855414032936096 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14971409738063812 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1974206417798996 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.32678577303886414 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10406121611595154 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09923860430717468 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9649305939674377 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8085107803344727 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11396247893571854 (particle/accuracy‚0.8397095203399658 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.09130623191595078 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.21388891339302063 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2780296206474304 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_4_of_6‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.31587299704551697 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.25555965304374695 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666865348816 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.10203321278095245 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10883455723524094 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2732612192630768 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_4‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.44516006112098694 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.004574435763061047 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.2745627462863922 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3541666865348816 +jet/accuracy_2_of_2‚0.07258052378892899 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_1_of_6‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.37832432985305786 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3988969027996063 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.2951677739620209 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666865348816 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.2640615403652191 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9278959631919861 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.11940579116344452 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11167341470718384 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4547107517719269 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3916666805744171 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.42495426535606384 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.19907408952713013 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2037356197834015 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2712481617927551 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.20039718342013¢¯h†Q7e-05 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2640015184879303 +jet/accuracy_3_of_5‚0.07587044686079025 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06465769559144974 jet/accuracy_4_of_7‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 +jet/accuracy_2_of_6‚0.26923078298568726 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20752489566802979 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9880821108818054 particle/accuracy_2_of_4‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.22814872860908508 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_7‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.990868091583252 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9705459475517273 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.015624498948454857 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5263158082962036 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09770115464925766 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2808642089366913 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6542194485664368 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2488303929567337 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 jet/accuracy_3_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.279077410697937 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.15354938805103302 particle/accuracy_4_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.34090909361839294 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 epoch‚2 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.01923076994717121 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 _runtime‚3508.471294062 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15419621765613556 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3904082477092743 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1523013561964035 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0015873016091063619 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9935672283172607 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10836552083492279 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9969996213912964 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.983015239238739 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9775568842887878 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.15102705359458923 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.06986524909734726 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9879782199859619 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.28125 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.12808440625667572 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25619176030158997 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_5_of_5‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.31609678268432617 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3956199586391449 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1677570939064026 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.27536100149154663 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_7‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013378684408962727 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1740325391292572 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2291666716337204 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.2650268077850342 'particle/f_score‚0.3403492271900177 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9949675798416138 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9717102646827698 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_3‚0.1712513566017151 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15176813304424286 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5124179124832153 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 _step‚296 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 trainer/global_step‚3626 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30173608660697937 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.31135058403015137 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2698003053665161 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05“ÿ†+¡`Á¥¥ ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.009803921915590763 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9733333587646484 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.40468207001686096 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2578543722629547 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11786902695894241 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9914483428001404 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.42596161365509033 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 )loss/val_total_loss‚48.62661683198177 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.17968539893627167 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.30106136202812195 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.028368795290589333 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9903141260147095 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.023418273776769638 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11453457921743393 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13494668900966644 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.33666664361953735 jet/accuracy_6_of_7‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3266666829586029 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0022338533308357 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_6‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16996479034423828 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.2948477268218994 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.991067111492157 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2482256144285202 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12573380768299103 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.2903914451599121 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.277045339345932 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.01360864844173193 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4208333492279053 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.381944477558136 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.01810620166361332 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 particle/accuracy_3_of_7‚1 particle/accuracy_7_of_7‚1 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.26068660616874695 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.33624032139778137 )jet/accuracy_1_of_2‚0.607465922832489 jet/accuracy_7_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9785714745521545 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 jet/accuracy_5_of_6‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.30000001192092896 jet/accuracy_2_of_7‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 *jet/accuracy_2_of_5‚0.3837749660015106 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9460555911064148 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3769063651561737 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.29973456263542175 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008818342466838658 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1845327615737915 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4769168794155121 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.268627405166626 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.36170637607574463 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1644144058227539 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7327113747596741 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4208090007305145 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1018165871500969 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12729884684085846 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.0768401250243187 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4120071530342102 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9954621195793152 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9845237135887146 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.27470239996910095 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09433633834123611 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.38864102959632874 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 <"Purity/0FRt2SRqqt1FBt/event_purity‚0.0026954179629683495 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2588961720466614 particle/accuracy_5_of_6‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2645302712917328 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 particle/accuracy_3_of_3‚1 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 +jet/accuracy_2_of_4‚0.23339654505252838 % _timestamp‚1.7884200377192292e+09 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09813167154788971 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2742424011230469 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_7‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.005708481650799513 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10181235522031784 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04412972182035446 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09912828356027603 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2803189754486084 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2770157754421234 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15914630889892578 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11058702319860458 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6217588186264038 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.006236972752958536 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.27619048953056335 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2657490074634552 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3343355059623718 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.47467362880706787 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04046492651104927 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10859786719083786 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3564814627170563 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2629435360431671 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.259869784116745 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.32466843724250793 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2423035055398941 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11385779082775116 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22636811435222626 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.255495548248291 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.26747533679008484 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2621200382709503 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 particle/accuracy_2_of_7‚1 *"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9885181188583374 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.38871318101882935 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9855414032936096 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14971409738063812 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1974206417798996 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.32678577303886414 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10406121611595154 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09923860430717468 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9649305939674377 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8085107803344727 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11396247893571854 (particle/accuracy‚0.8397095203399658 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.09130623191595078 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.21388891339302063 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2780296206474304 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_4_of_6‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.31587299704551697 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.25555965304374695 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666865348816 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.10203321278095245 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10883455723524094 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2732612192630768 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_4‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.44516006112098694 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.004574435763061047 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.2745627462863922 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3541666865348816 +jet/accuracy_2_of_2‚0.07258052378892899 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_1_of_6‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.37832432985305786 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3988969027996063 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.2951677739620209 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666865348816 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.2640615403652191 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9278959631919861 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.11940579116344452 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11167341470718384 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4547107517719269 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3916666805744171 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.42495426535606384 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.19907408952713013 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2037356197834015 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2712481617927551 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2640015184879303 +jet/accuracy_3_of_5‚0.07587044686079025 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06465769559144974 jet/accuracy_4_of_7‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 +jet/accuracy_2_of_6‚0.26923078298568726 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20752489566802979 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9880821108818054 particle/accuracy_2_of_4‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.22814872860908508 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_7‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.990868091583252 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9705459475517273 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.015624498948454857 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5263158082962036 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09770115464925766 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2808642089366913 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6542194485664368 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2488303929567337 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 jet/accuracy_3_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.279077410697937 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.15354938805103302 particle/accuracy_4_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.34090909361839294 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 epoch‚2 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.01923076994717121 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 _runtime‚3508.471294062 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15419621765613556 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3904082477092743 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1523013561964035 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRgÞ™ 2t4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0015873016091063619 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9935672283172607 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10836552083492279 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9969996213912964 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.983015239238739 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9775568842887878 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.15102705359458923 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.06986524909734726 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9879782199859619 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.28125 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.12808440625667572 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25619176030158997 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_5_of_5‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.31609678268432617 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3956199586391449 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1677570939064026 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.27536100149154663 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_7‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013378684408962727 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1740325391292572 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2291666716337204 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.2650268077850342 'particle/f_score‚0.3403492271900177 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9949675798416138 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9717102646827698 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_3‚0.1712513566017151 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15176813304424286 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5124179124832153 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 _step‚296 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 trainer/global_step‚3626 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30173608660697937 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.31135058403015137 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2698003053665161 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3709937632083893 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3586309552192688 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 5validation_average_jet_accuracy‚0.341006338596344 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3991319239139557 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5075757503509521 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0317460335791111 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 -jet/accuracy_4_of_4‚0.0008818342466838658 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636362552642822 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.987565815448761 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15611879527568817 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3035714328289032 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02467358112335205 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.21604938805103302 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2777256667613983 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.37176090478897095 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9897756576538086 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9900497198104858 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3074030876159668 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0297619067132473 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.810495913028717 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2361111044883728 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.008543875999748707 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 )jet/accuracy_1_of_5‚0.879621148109436 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07999999821186066 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0968480259180069 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.992092490196228 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.30778229236602783 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10905350744724274 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_6‚0.9294872283935547 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.006593769416213036 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05480670556426048 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.25585582852363586 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17676766216754913 particle/accuracy_1_of_5‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31827235221862793 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_5_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2923413813114166 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.03270745277404785 particle/accuracy_2_of_3‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.2903914451599121 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2777777910232544 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11198775470256805 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5075757503509521 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12829822301864624 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.09130623191595078 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10818713158369064 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.31609195470809937 particle/accuracy_1_of_4‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.170929417014122 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9876256585121155 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2687363624572754 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9883720278739929¨oîÅؙ‚(:Ž Ç¿äÔ‰´¸x gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚29.0 gpu.0.powerWatts‚76.035' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.206999999999999 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛrµuÂ(:ê Ç¿äÔ§¾±{ disk.sda3.in‚7.3125 disk.rbd1.in‚0 disk.dm-1.out‚223.375& proc.memory.availableMB‚ 6.50390625* proc.memory.percent‚18.431735038757324 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda2.in‚0 cpu‚25.301934920565653) disk./.usagePercent‚73.71480480101374& disk.nvme0n1.out‚14871.54638671875 disk.drbd1777.out‚0# disk.dm-0.out‚14871.54638671875 disk.sda2.out‚ 0.0390625 proc.cpu.threads‚17 disk.sda.out‚ 223.4140625 disk.sda3.out‚223.375 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 network.sent‚ 7.943926e+06 network.recv‚ 1.226097e+06 disk.rbd1.out‚ 1.59619140625 disk.dm-0.in‚ 4.9296875 disk.dm-1.in‚7.3125 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9296875( disk.nvme0c0n1.out‚14871.54638671875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 4.9296875 disk.rbd2.in‚0! disk.rbd2.out‚135.93408203125$ memory_percent‚99.96030330657959# proc.memory.rssMB‚ 3019.85546875 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.rbd4.in‚0% disk./.usageGB‚1098.4038619995117é`$Ä‚(:… Ö¿äÔçó®x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.867' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9734 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ ßqÅ‚(:æ Ö¿äÔ¿®| network.recv‚ 1.232747e+06 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.out‚0 disk.sda.out‚ 224.109375 disk.rbd1.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0* proc.memory.percent‚17.399215698242188( disk./.usagePercent‚73.7105965491659' disk.nvme0c0n1.out‚15137.2587890625 disk.sda3.in‚7.3125 disk.nvme0n1.in‚ 4.9296875 disk.sda2.out‚ 0.0390625 disk.dm-0.in‚ 4.9296875 disk.dm-1.out‚ 224.0703125 disk.rbd0.out‚0$ memory_percent‚97.91183471679688 proc.cpu.threads‚17 disk.sda.in‚ 7.36328125 disk.rbd2.in‚0% disk./.usageGB‚1098.3411560058594 disk.sda3.out‚ 224.0703125" disk.dm-0.out‚15137.2587890625! disk.rbd2.out‚137.44580078125# proc.memory.availableMB‚342.125 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 proc.memory.rssMB‚ 2850.6875 cpu‚25.300715581722635 disk.rbd1.out‚ 1.59619140625 disk.drbd1777.out‚0 network.sent‚ 8.068344e+06 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0% disk.nvme0n1.out‚15137.2587890625 disk.dm-1.in‚7.3125 disk.rbd4.out‚0qýÅ(2Æ- _wandbruntime‚3531  _runtime‚3531ÎR¼ÅÇ¿ % lr-AdamW/pg1‚0.000706821124101469 % lr-AdamW/pg2‚0.000706821124101469 trainer/global_step‚3649 % _timestamp‚1.7884200610815063e+09 _runtime‚3531.831453432 _step‚297Üņ+ÊÈÄ % lr-AdamW/pg2‚0.000706821124101469 trainer/global_step‚3649 % _timestamp‚1.7884200610815063e+09 _runtime‚3531.831453432 _step‚297 % lr-AdamW/pg1‚0.000706821124101469©”QH“2É- _wandbruntime‚3531  _runtime‚3531È`ñÛÛ'ÊÕO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95063678 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 % _timestamp‚1.7884200610822325e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚3531.832370199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64080721 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚144891152151552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6781819449856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚151672971601408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162573085 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67509409 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 trainer/global_step‚3649 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚162369223008768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6781819449856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67509407 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚151672971601408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144891170154496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6762129227966 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64080043 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162369524158976 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143957611250878 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6903905423872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155465317584896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39501216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24579503 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚144891170154496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95089348 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95063678 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95089348 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚151673217807872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67509409 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39501218 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6762352877114 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561ûº“ÊËËÚO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6781819449856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67509407 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚151672971601408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144891170154496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6762129227966 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64080043 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162369524158976 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143957611250878 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6903905423872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155465317584896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39501216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24579503 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚144891170154496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95089348 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95063678 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95089348 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚151673217807872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67509409 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39501218 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6762352877114 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95063678 N?DeviceStatsMonitor.on_traimÞ¿n_batch_start/reserved_bytes.all.peak‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 % _timestamp‚1.7884200610822325e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚3531.832370199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64080721 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚144891152151552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6781819449856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚151672971601408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162573085 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67509409 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 trainer/global_step‚3649 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚162369223008768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299ªxÔCz2Ì- _wandbruntime‚3532  _runtime‚3532˪vå&ÍßM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137231784480768 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67527126 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162642053 _runtime‚3532.695010715 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67527128 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39511571 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95114925 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95089229 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6764185276922 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67527128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6783657144320 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 % _timestamp‚1.7884200619442458e+09 trainer/global_step‚3649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6783885361152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95089229 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155506511221248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚155506487281152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚144929508098560 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6906042439168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 260419584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6905782019584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162616355 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6783657144320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143995986797050 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39511573 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚137231801520128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144929490095616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24585645 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6763961627566 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚144929490095616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162642053 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162616355 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95114925 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64096581 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143995746108334 _step‚299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚151713393459712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635à\upê&ÎäM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39511573 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚137231801520128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144929490095616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24585645 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6763961627566 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚144929490095616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162642053 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162616355 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95114925 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64096581 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143995746108334 _step‚299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚151713393459712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137231784480768 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67527126 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162642053 _runtime‚3532.695010715 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67527128 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39511571 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95114925 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95089229 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6764185276922 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67527128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6783657144320 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 % _timestamp‚1.7884200619442458e+09 trainer/global_step‚3649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6783885361152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95089229 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155506511221248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚155506487281152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚144929508098560 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6906042439168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 260419584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6905782019584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162616355 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6783657144320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143995986797050 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561« £Ú02Ï- _wandbruntime‚3532  _runtime‚3532ÑídìZÐÔ 2loss/SRqqt2/assignment_loss‚1.3065037466585636 1loss/FBt2/assignment_loss‚0.22934305210473002 /loss/FRt4/detection_loss‚2.2308455952938577 0loss/FBt1/assignment_loss‚0.7624822475693442 1loss/FBt4/assignment_loss‚0.34268808568065817 .loss/FBt3/detection_loss‚4.516931891441345 0loss/SRqqt3/detection_loss‚1.494927894888502 1loss/SRqqt4/detection_loss‚1.8011730467279752 /loss/FBt2/detection_loss‚0.5833963563579113 1loss/SRqqt2/detection_loss‚1.0418630954535568 epoch‚3 _step‚300 2loss/SRqqt4/assignment_loss‚2.1830887814362843 /loss/FRt1/detection_loss‚2.5154826011508704 0loss/FRt2/assignment_loss‚2.0896103580792746 0loss/FRt3/assignment_loss‚3.7794664025306703 % _timestamp‚1.7884200619461384e+09 /loss/FRt1/assignment_loss‚3.828682582080364 /loss/FRt4/assignment_loss‚4.582433320106344 0loss/FBt3/assignment_loss‚0.4416493936441839 /loss/FRt3/detection_loss‚2.2453549313545227 1loss/SRqqt1/detection_loss‚2.1908808625058125 /loss/FBt1/detection_loss‚2.6378335004503075 %loss/total_loss‚49.45609577752617 trainer/global_step‚3649 1loss/SRqqt1/assignment_loss‚2.149406069203427 2loss/SRqqt3/assignment_loss‚1.8993439319910426 .loss/FRt2/detection_loss‚1.736603165200601 .loss/FBt4/detection_loss‚2.866104865616018 _runtime‚3532.696008304"uœÎ_ÑÙ 1loss/SRqqt2/detection_loss‚1.0418630954535568 epoch‚3 _step‚300 2loss/SRqqt4/assignment_loss‚2.1830887814362843 /loss/FRt1/detection_loss‚2.5154826011508704 0loss/FRt2/assignment_loss‚2.0896103580792746 0loss/FRt3/assignment_loss‚3.7794664025306703 % _timestamp‚1.7884200619461384e+09 /loss/FRt1/assignment_loss‚3.828682582080364 /loss/FRt4/assignment_loss‚4.582433320106344 0loss/FBt3/assignment_loss‚0.4416493936441839 /loss/FRt3/detection_loss‚2.2453549313545227 1loss/SRqqt1/detection_loss‚2.1908808625058125 /loss/FBt1/detection_loss‚2.6378335004503075 %loss/total_loss‚49.45609577752617 trainer/global_step‚3649 1loss/SRqqt1/assignment_loss‚2.149406069203427 2loss/SRqqt3/assignment_loss‚1.8993439319910426 .loss/FRt2/detection_loss‚1.736603165200601 .loss/FBt4/detection_loss‚2.866104865616018 _runtime‚3532.696008304 2loss/SRqqt2/assignment_loss‚1.3065037466585636 1loss/FBt2/assignment_loss‚0.22934305210473002 /loss/FRt4/detection_loss‚2.2308455952938577 0loss/FBt1/assignment_loss‚0.7624822475693442 1loss/FBt4/assignment_loss‚0.34268808568065817 .loss/FBt3/detection_loss‚4.516931891441345 0loss/SRqqt3/detection_loss‚1.494927894888502 1loss/SRqqt4/detection_loss‚1.8011730467279752 /loss/FBt2/detection_loss‚0.5833963563579113¬oèazŽÒ‚(:ƒ å¿äÔÑŽ¨x gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òØu©zÓ‚(:ï å¿äÔáíùz" disk.dm-0.out‚15304.1806640625 disk.sda2.in‚0' disk.nvme0c0n1.out‚15304.1806640625 proc.memory.rssMB‚ 2850.6875 disk.dm-1.out‚ 224.43359375# memory_percent‚97.9132890701294( proc.memory.availableMB‚ 341.88671875 proc.cpu.threads‚17) disk./.usagePercent‚73.71055072381726% disk./.usageGB‚1098.3404731750488 disk.nvme0n1.in‚ 4.9296875! disk.rbd2.out‚137.58642578125 disk.rbd4.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.299681847728976 disk.sda3.out‚ 224.43359375 disk.dm-1.in‚7.3125 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125* proc.memory.percent‚17.399215698242188 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0 network.sent‚ 8.074695e+06 disk.sda.in‚ 7.36328125 disk.sda.out‚ 224.47265625 disk.rbd1.in‚0 disk.dm-0.in‚ 4.9296875 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 network.recv‚ 1.237388e+06 disk.sda3.in‚7.3125% disk.nvme0n1.out‚15304.1806640625 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd1.out‚ 1.68212890625CdØaŽÔ‚(:ƒ ô¿äÔÞˆ’x gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$6ªpÕ‚(:å ô¿äÔ ÔÁ{! disk.rbd2.out‚137.58642578125 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 225.2734375 network.sent‚ 8.18283e+06% disk./.usageGB‚1098.3432502746582 disk.rbd4.out‚0' disk.nvme0c0n1.out‚15406.4013671875% proc.memory.availableMB‚ 341.78125 proc.cpu.threads‚17 network.recv‚ 1.243466e+06 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda.out‚225.3125 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda3.in‚7.3125 disk.rbd1.out‚ 1.68212890625 disk.dm-0.in‚ 4.9296875" disk.dm-0.out‚15406.4013671875 disk.drbd1777.in‚ 1.03515625 proc.memory.rssMB‚ 2850.6875) disk./.usagePercent‚73.71073709730219 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9296875 disk.rbd0.out‚0$ memory_percent‚97.91393280029297* proc.memory.percent‚17.399215698242188% disk.nvme0n1.out‚15406.4013671875 disk.rbd3.in‚0 cpu‚25.29862805433736 disk.dm-1.out‚ 225.2734375 disk.rbd2.in‚0ôÔq‹Ö‚(:€ ƒÀäÔðž™x gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.5' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#Órׂ(:ç ƒÀäÔÔ§Áz cpu‚25.2975361074554 network.sent‚ 8.188894e+06% disk./.usageGB‚1098.3432502746582! disk.dm-0.out‚15505.482421875 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.71073709730219 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 225.80078125 disk.rbd1.in‚0$ memory_percent‚97.91393280029297 disk.dm-0.in‚ 4.9296875 disk.dm-1.in‚7.3125 proc.memory.rssMB‚ 2850.6875 disk.sda2.in‚0 disk.dm-1.out‚ 225.76171875 disk.rbd2.out‚137.7548828125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda1.out‚0 network.recv‚ 1.247724e+06& disk.nvme0c0n1.out‚15505.482421875 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.nvme0n1.in‚ 4.9296875$ disk.nvme0n1.out‚15505.482421875% proc.memory.availableMB‚ 341.78125 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 225.76171875 disk.rbd1.out‚ 1.74462890625* proc.memory.percent‚17.399215698242188uP2Ø- _wandbruntime‚3576  _runtime‚3576Ž ³ÂÇÙÁ % _timestamp‚1.7884201062332265e+09 _runtime‚3576.983152268 _step‚301 & lr-AdamW/pg1‚0.0007030387944040435 & lr-AdamW/pg2‚0.0007030387944040435 trainer/global_step‚3699h¦ócÌÚÆ % _timestamp‚1.7884201062332265e+09 _runtime‚3576.983152268 _step‚301 & lr-AdamW/pg1‚0.0007030387944040435 & lr-AdamW/pg2‚0.0007030387944040435 trainer/global_step‚3699­'ƒÇ2Û- _wandbruntime‚3576  _runtime‚35763– ˜‡ÜÕO % _timestamp‚1.7884201062339506e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157523802397696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 trainer/globa öO˜T%l_step‚3699 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145864367714886 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96344580 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64921804 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96370250 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64922487 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153681767941632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153682014148096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6873946790400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40018968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6997329216000 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146808067357696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40018966 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6853762799174 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145864608403394 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚139010604915712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24902838 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚302 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96344580 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164521131613696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6997606426112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68395359 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146808049354752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6873718586880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6873946790400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164521432763904 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚139010621955072 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96370250 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153682014148096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6997329216000 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146808067357696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40018966 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6853762799174 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145864608403394 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚139010604915712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24902838 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚302 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96344580 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164521131613696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6997606426112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68395359 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146808049354752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6873718586880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6873946790400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164521432763904 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚139010621955072 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96370250 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153682014148096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153681767941632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153682014148096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6873946790400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40018968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6875784498176 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164783207 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68413078 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153721943580160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164564465515008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40029321 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157564972093952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153722189799936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146846387298816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139046924412928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 trainer/global_step‚3699 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40029323 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6999469480960 _step‚303 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6855818848130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64938466 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164808905 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292748288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153722189799936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164783207 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3577.855932849 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96370131 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164564172766720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6875784498176 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164783207 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68413078 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153721943580160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164564465515008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4002 Ë6hX9321 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157564972093952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153722189799936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146846387298816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139046924412928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 trainer/global_step‚3699 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40029323 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6999469480960 _step‚303 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6855818848130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64938466 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164808905 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292748288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153722189799936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164783207 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3577.855932849 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96370131 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164564172766720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159582311150592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159582287210496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884201514936292e+09 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7090726846976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚155690810488320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 _step‚306 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166932461 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97625482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 309538816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚140825727808512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6945620019530 trainer/global_step‚3749 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚148724964560896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65764766 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7091012445696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 285598720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25228048 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 166906789 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166673014057472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148724964560896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69281307 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155690564281856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6965617723904 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97625482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 309538816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚140825727808512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6945620019530 trainer/global_step‚3749 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚148724964560896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65764766 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7091012445696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 285598720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25228048 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 166906789 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166673014057472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148724964560896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69281307 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155690564281856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6965617723904 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159582311150592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159582287210496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884201514936292e+09 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7090726846976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚155690810488320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 _step‚306²9ÇyV2ð- _wandbruntime‚3623  _runtime‚3623A Rä&ñÞM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159623456906752 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40547071 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _step‚307 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166950059 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69299026 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7092868204032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚140862030266368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69299026 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6967455418368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65781609 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚148763284502016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6947228769982 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166975757 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚155730739920384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚155730986140160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97676729 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69299028 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25233889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148763284502016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69299028 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65780960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚147809499725066 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 $ _timestamp‚1.788420152354075e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166975757 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 _runtime‚3623.104668372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25234536 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166950059 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6947452419338 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚155730986140160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _step‚307 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166950059 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69299026 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7092868204032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚140862030266368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69299026 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6967455418368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65781609 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚148763284502016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6947228769982 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166975757 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚155730739920384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚155730986140160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97676729 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69299028 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25233889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148763284502016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69299028 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65780960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚147809499725066 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 $ _timestamp‚1.788420152354075e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166975757 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 _runtime‚3623.104668372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25234536 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166950059 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6947452419338 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚155730986140160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157699360622080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98932054 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161640795963392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚142640850701312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚157699606828544 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150641843761152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚149677880642902 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segmªèÈØÍent.all.peak‚561 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70167257 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150641861764096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚157699606828544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168825366055936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚150641861764096 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 % _timestamp‚1.7884201966845617e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25548534 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7057745064448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7184294979584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169099313 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚142640867740672 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98932054 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149678121331410 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7037253590738 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169073641 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70167259 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7057516860928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7184570092544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7057516860928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70167259 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _runtime‚3667.434699098 _step‚310 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7037029941590 trainer/global_step‚3799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168825067002880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70167257 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41054468 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25549195 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262/ÀÀà'ÚO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7037029941590 trainer/global_step‚3799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168825067002880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70167257 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41054468 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25549195 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98906384 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98906384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7057745064448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157699360622080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98932054 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161640795963392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚142640850701312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚157699606828544 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150641843761152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚149677880642902 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70167257 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150641861764096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚157699606828544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168825366055936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚150641861764096 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 % _timestamp‚1.7884201966845617e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25548534 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7057745064448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7184294979584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169099313 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚142640867740672 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98932054 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149678121331410 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7037253590738 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169073641 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70167259 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7057516860928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7184570092544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7057516860928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70167259 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _runtime‚3667.434699098 _step‚310¶<éá2‚- _wandbruntime‚3668  _runtime‚3668µf.9å&ƒßM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7059582772224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7038862341190 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _runtime‚3668.301667669 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚157739782480384 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150680181705216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98931935 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚142677153159168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98931935 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7059582772224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7186173311488 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 _step‚311 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169142609 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66620827 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150680181705216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169116911 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150680199708160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚157739536260608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66620205 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169142609 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70184978 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41064821 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚157739782480384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161681941719552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 % _timestamp‚1.7884201975512156e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150680199708160 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70184976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98957631 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168868397293568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161681965659648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 trainer/global_step‚3799 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98957631 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25555384 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150680199708160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚157739536260608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66620205 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169142609 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70184978 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41064821 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚157739782480384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161681941719552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 % _timestamp‚1.7884201975512156e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150680199708160 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70184976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98957631 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168868397293568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161681965659648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 trainer/global_step‚3799 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98957631 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25555384 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7059582772224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7038862341190 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _runtime‚3668.301667669 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚157739782480384 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150680181705216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98931935 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚142677153159168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98931935 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7059582772224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7186173311488 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 _step‚311 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169142609 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66620827 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150680181705216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832·Ç(‹2…- _wandbruntime‚3668  _runtime‚3668ŠzÚT\†Ö 0loss/FBt2/assignment_loss‚0.2232201399003235 1loss/SRqqt3/detection_loss‚2.3478784633752627 1loss/SRqqt4/detection_loss‚1.8821137416362763 /loss/FRt1/assignment_loss‚4.296163085030346 /loss/FRt3/assignment_loss‚3.751381078833028 .loss/FBt1/detection_loss‚2.301290596524874 epoch‚3 .loss/FRt1/detection_loss‚2.429220833429476 .loss/FRt3/detection_loss‚2.798675204578199 /loss/FRt4/detection_loss‚2.4429865138871327 2loss/SRqqt2/assignment_loss‚1.1739703520913334 2loss/SRqqt4/assignment_loss‚1.8663871258497238 1loss/FBt3/assignment_loss‚0.15069707052310832 /loss/FBt4/detection_loss‚3.3161032464769153 0loss/FRt2/assignment_loss‚1.7384864552965704 2loss/SRqqt1/assignment_loss‚2.1628903735961234 .loss/FBt3/detection_loss‚3.532633145650228 %loss/total_loss‚49.17341161544867 % _timestamp‚1.7884201975530517e+09 _step‚312 1loss/SRqqt2/detection_loss‚1.1726675239555977 0loss/FRt4/assignment_loss‚3.7726275614329747 2loss/SRqqt3/assignment_loss‚1.5046101311842601 1loss/FBt4/assignment_loss‚0.44605452277594143 1loss/SRqqt1/detection_loss‚2.4489921650716235 0loss/FBt2/detection_loss‚0.45283098429855373 _runtime‚3668.302904148 0loss/FBt1/assignment_loss‚0.9733774724105521 /loss/FRt2/detection_loss‚1.9881538276402455 trainer/global_step‚3799Ll&ta‡Û 1loss/SRqqt2/detection_loss‚1.1726675239555977 0loss/FRt4/assignment_loss‚3.7726275614329747 2loss/SRqqt3/assignment_loss‚1.5046101311842601 1loss/FBt4/assignment_loss‚0.44605452277594143 1loss/SRqqt1/detection_loss‚2.4489921650716235 0loss/FBt2/detection_loss‚0.45283098429855373 _runtime‚3668.302904148 0loss/FBt1/assignment_loss‚0.9733774724105521 /loss/FRt2/detection_loss‚1.9881538276402455 trainer/global_step‚3799 0loss/FBt2/assignment_loss‚0.2232201399003235 1loss/SRqqt3/detection_loss‚2.3478784633752627 1loss/SRqqt4/detection_loss‚1.8821137416362763 /loss/FRt1/assignment_loss‚4.296163085030346 /loss/FRt3/assignment_loss‚3.751381078833028 .loss/FBt1/detection_loss‚2.301290596524874 epoch‚3 .loss/FRt1/detection_loss‚2.429220833429476 .loss/FRt3/detection_loss‚2.798675204578199 /loss/FRt4/detection_loss‚2.4429865138871327 2loss/SRqqt2/assignment_loss‚1.1739703520913334 2loss/SRqqt4/assignment_loss‚1.8663871258497238 1loss/FBt3/assignment_loss‚0.15069707052310832 /loss/FBt4/detection_loss‚3.3161032464769153 0loss/FRt2/assignment_loss‚1.7384864552965704 2loss/SRqqt1/assignment_loss‚2.1628903735961234 .loss/FBt3/detection_loss‚3.532633145650228 %loss/total_loss‚49.17341161544867 % _timestamp‚1.7884201975530517e+09 _step‚312¸K@-2ˆ‚(:… ìÀäÔ£•Àx gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ ¢ëu‰‚(:ê ìÀäÔú´ðz& disk.nvme0n1.out‚16003.41552734375# disk.dm-0.out‚16003.41552734375 network.sent‚ 8.443879e+06 network.recv‚ 1.296606e+06$ disk./.usageGB‚1098.343978881836 disk.nvme0n1.in‚ 4.9296875* proc.memory.percent‚17.399215698242188 disk.nvme0c0n1.in‚ 4.9296875$ memory_percent‚98.93598556518555 proc.cpu.threads‚17 disk.sda.out‚ 229.828125! disk.rbd2.out‚139.52880859375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0) disk./.usagePercent‚73.71078599474123 disk.rbd1.in‚0 disk.dm-1.out‚ 229.7890625 disk.drbd1777.out‚0 disk.sda.in‚ 7.36328125 disk.dm-0.in‚ 4.9296875 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125& proc.memory.availableMB‚ 174.328125 disk.sda3.out‚ 229.7890625 disk.rbd1.out‚ 1.90283203125 disk.sda2.out‚ 0.0390625 disk.rbd4.out‚0( disk.nvme0c0n1.out‚16003.41552734375 proc.memory.rssMB‚ 2850.6875 disk.sda3.in‚7.3125 disk.rbd3.in‚0 disk.rbd3.out‚0 cpu‚25.2903063209944öOW"Š‚(:… ûÀäÔúò‚x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i Qz‹‚(:ï ûÀäÔ™×¹{ network.sent‚ 8.552936e+06 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.399215698242188 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625 disk.sda3.out‚ 230.16796875& disk.nvme0n1.out‚16102.50048828125# disk.dm-0.out‚16102.50048828125$ disk./.usageGB‚1098.343978881836 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda3.in‚7.3125 proc.memory.rssMB‚ 2850.6875 cpu‚25.28932853127443 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 230.16796875 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚16102.50048828125) disk./.usagePercent‚73.71078599474123 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.dm-0.in‚ 4.9296875$ memory_percent‚98.93598556518555 network.recv‚ 1.303847e+06! disk.rbd2.out‚141.18115234375 disk.nvme0c0n1.in‚ 4.9296875 disk.sda.out‚ 230.20703125 disk.rbd1.out‚ 1.90283203125 disk.nvme0n1.in‚ 4.9296875& proc.memory.availableMB‚ 174.328125JîÔ|Œ‚(:„ ŠÁäÔ¡›x gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.623 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¢‡v‚(:ë ŠÁäÔåÐÛz network.recv‚ 1.308254e+06 disk.sda2.out‚ 0.0390625 disk.rbd3.out‚0 disk.rbd1.out‚ 1.90283203125 proc.cpu.threads‚17 disk.rbd4.out‚0 disk.rbd2.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚16103.14794921875 disk.sda3.out‚ 230.734375( proc.memory.availableMB‚ 174.08203125 cpu‚25.28837673097755! disk.rbd2.out‚141.18115234375 disk.dm-0.in‚ 4.9296875 disk.sda3.in‚7.3125 proc.memory.rssMB‚ 2850.6875& disk.nvme0n1.out‚16103.14794921875 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9296875) disk./.usagePercent‚73.71078599474123# disk.dm-0.out‚16103.14794921875 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.in‚0 disk.rbd1.in‚0$ memory_percent‚98.93748760223389$ disk./.usageGB‚1098.343978881836 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0* proc.memory.percent‚17.399215698242188 network.sent‚ 8.55937e+06 disk.nvme0n1.in‚ 4.9296875 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 230.734375 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 230.7734375åeyv2Ž- _wandbruntime‚3712  _runtime‚3712’ºXÇÁ % _timestamp‚1.7884202419436023e+09 _runtime‚3712.693535522 _step‚313 & lr-AdamW/pg1‚0.0006913175562645831 & lr-AdamW/pg2‚0.0006913175562645831 trainer/global_step‚3849‹(TàÌÆ _runtime‚3712.693535522 _step‚313 & lr-AdamW/pg1‚0.0006913175562645831 & lr-AdamW/pg2‚0.0006913175562645831 trainer/global_step‚3849 % _timestamp‚1.7884202419436023e+09¹}KU2‘- _wandbruntime‚3712  _runtime‚3712Ò¶ìoß'’ÙO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚152558758967296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41572216 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71053207 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚152558758967296 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151584877795418 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159708403168768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7149644201472 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3849 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7277871228416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71053209 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67444284 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170977152004608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151584637106910 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71053207 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _step‚314 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159708156962304 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163699280776192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7128887161946 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7128663512798 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7149644201472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163699256836096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159708156962304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7277591921152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41572218 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171266165 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100212956 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100187286 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159708403168768 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100212956 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7149415997952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚152558740964352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884202419444227e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171240493 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3849 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7277871228416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71053209 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67444284 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170977152004608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151584637106910 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71053207 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _step‚314 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159708156962304 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163699280776192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7128887161946 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7128663512798 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7149644201472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163699256836096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159708156962304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7277591921152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41572218 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171266165 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100212956 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100187286 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159708403168768 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100212956 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7149415997952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚152558740964352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884202419444227e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171240493 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚152558758967296 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151584877795418 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WIDeviceStatsMonitor.on_train_lxCaþbatch_start/reserved_bytes.small_pool.current‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159708403168768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7149644201472 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67461705 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71070928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100238533 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163740450472448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100212837 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7130719561754 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7151481909248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67461056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚152597078908416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚152597096911360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71070926 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚152597096911360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚3849 % _timestamp‚1.7884202428118315e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171283763 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100212837 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41582573 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚152597078908416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71070926 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7279465114624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7151481909248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159748578820608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171309461 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171309461 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144492293091328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7151253692416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171283763 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7151253692416 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67461056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚152597078908416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚152597096911360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71070926 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚152597096911360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚3849 % _timestamp‚1.7884202428118315e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171283763 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100212837 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41582573 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚152597078908416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71070926 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7279465114624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7151481909248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159748578820608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171309461 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171309461 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144492293091328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7151253692416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171283763 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7151253692416 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67461705 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71070928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100238533 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163740450472448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0»EðÒ2—- _wandbruntime‚3713  _runtime‚3713òÝù9Y˜Ó /loss/FRt1/assignment_loss‚4.564464630502643 /loss/FRt3/assignment_loss‚3.621275473863651 0loss/FRt4/assignment_loss‚4.0029007622173856 1loss/SRqqt4/detection_loss‚2.1215082454410465 2loss/SRqqt4/assignment_loss‚1.9655350090946409 /loss/FBt3/assignment_loss‚0.501095088198781 /loss/FRt3/detection_loss‚2.6573579143255186 $ _timestamp‚1.788420242813798e+09 2loss/SRqqt1/assignment_loss‚1.8242758259177208 /loss/FBt4/detection_loss‚2.7306311666965484 %loss/total_loss‚48.68083901129714 trainer/global_step‚3849 1loss/SRqqt2/detection_loss‚0.9769884499151316 1loss/SRqqt3/detection_loss‚1.7965465257013287 epoch‚3 _runtime‚3713.563602582 /loss/FRt1/detection_loss‚2.5721249466366842 2loss/SRqqt3/assignment_loss‚1.8263386078178883 1loss/FBt2/assignment_loss‚0.27007221826290445 /loss/FRt2/assignment_loss‚2.033566140466266 .loss/FRt4/detection_loss‚2.395716532355263 0loss/SRqqt1/detection_loss‚2.114553038775921 _step‚316 0loss/FBt4/assignment_loss‚0.7215019958092853 2loss/SRqqt2/assignment_loss‚1.1575972682504512 0loss/FBt1/assignment_loss‚0.8921283998256513 /loss/FBt3/detection_loss‚3.2681517601013184 /loss/FRt2/detection_loss‚1.6704169986424622 /loss/FBt1/detection_loss‚2.2896541086110203 /loss/FBt2/detection_loss‚0.70643790386762330~ò^™Ø 2loss/SRqqt3/assignment_loss‚1.8263386078178883 1loss/FBt2/assignment_loss‚0.27007221826290445 /loss/FRt2/assignment_loss‚2.033566140466266 .loss/FRt4/detection_loss‚2.395716532355263 0loss/SRqqt1/detection_loss‚2.114553038775921 _step‚316 0loss/FBt4/assignment_loss‚0.7215019958092853 2loss/SRqqt2/assignment_loss‚1.1575972682504512 0loss/FBt1/assignment_loss‚0.8921283998256513 /loss/FBt3/detection_loss‚3.2681517601013184 /loss/FRt2/detection_loss‚1.6704169986424622 /loss/FBt1/detection_loss‚2.2896541086110203 /loss/FBt2/detection_loss‚0.7064379038676233 /loss/FRt1/assignment_loss‚4.564464630502643 /loss/FRt3/assignment_loss‚3.621275473863651 0loss/FRt4/assignment_loss‚4.0029007622173856 1loss/SRqqt4/detection_loss‚2.1215082454410465 2loss/SRqqt4/assignment_loss‚1.9655350090946409 /loss/FBt3/assignment_loss‚0.501095088198781 /loss/FRt3/detection_loss‚2.6573579143255186 $ _timestamp‚1.788420242813798e+09 2loss/SRqqt1/assignment_loss‚1.8242758259177208 /loss/FBt4/detection_loss‚2.7306311666965484 %loss/total_loss‚48.68083901129714 trainer/global_step‚3849 1loss/SRqqt2/detection_loss‚0.9769884499151316 1loss/SRqqt3/detection_loss‚1.7965465257013287 epoch‚3 _runtime‚3713.563602582 /loss/FRt1/detection_loss‚2.5721249466366842¼ ®yÞš‚(:„ ™ÁäÔÀ³áy gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.261' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0522 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Dwž{›‚(:ð ™ÁäÔ™ŸÒz disk.sda.out‚ 231.22265625 disk.rbd4.out‚0 disk.sda.in‚ 7.36328125 disk.dm-1.in‚7.3125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚16202.17919921875 disk.rbd3.out‚0 disk.nvme0n1.in‚ 4.9296875 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 cpu‚25.287356526420474 disk.sda3.out‚ 231.18359375& disk.nvme0n1.out‚16202.17919921875 disk.dm-0.in‚ 4.9296875 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9296875* proc.memory.percent‚17.399215698242188 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 231.18359375 disk.rbd2.in‚0$ memory_percent‚98.93941879272461) disk./.usagePercent‚73.71078599474123 disk.sda3.in‚7.3125# disk.dm-0.out‚16202.17919921875! disk.rbd2.out‚141.25537109375 disk.drbd1777.out‚0& proc.memory.availableMB‚ 173.765625 proc.memory.rssMB‚ 2850.6875 disk.rbd1.out‚ 1.90283203125 network.sent‚ 8.576057e+06 network.recv‚ 1.333852e+06$ disk./.usageGB‚1098.343978881836 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0ˆ`óÖœ‚(:… ¨ÁäÔÐÞ•x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.892' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æW:}‚(:ò ¨ÁäÔ®—©z# disk.dm-0.out‚16301.92919921875 disk.dm-1.out‚ 231.93359375 cpu‚25.286414505665277 proc.cpu.threads‚17( disk.nvme0c0n1.out‚16301.92919921875 disk.sda3.in‚7.3125 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.90283203125 disk.rbd2.in‚0$ memory_percent‚98.93782138824463 network.sent‚ 8.682602e+06 disk.sda.in‚ 7.36328125 disk.sda.out‚ 231.97265625 proc.memory.rssMB‚ 2850.6875 network.recv‚ 1.338279e+06) disk./.usagePercent‚73.71078599474123 disk.sda1.out‚0 disk.dm-0.in‚ 4.9296875 disk.nvme0n1.in‚ 4.9296875& disk.nvme0n1.out‚16301.92919921875 disk.rbd0.out‚0( proc.memory.availableMB‚ 174.02734375 disk.rbd3.out‚0$ disk./.usageGB‚1098.343978881836 disk.rbd4.in‚0 disk.sda3.out‚ 231.93359375 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0* proc.memory.percent‚17.399215698242188 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9296875! disk.rbd2.out‚141.25537109375ù¥ôž‚(:„ ·ÁäÔÛµ™x gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚88.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GéV*qŸ‚(:æ ·ÁäÔƒ˜Ò{ network.sent‚ 8.694858e+06 disk.sda3.in‚7.3125 disk.sda2.in‚0 disk.dm-1.out‚232.3125 disk.rbd2.out‚ 141.361328125 disk.sda.in‚ 7.36328125# disk.dm-0.out‚16400.94091796875 disk.sda2.out‚ 0.0390625 disk.rbd3.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚73.71287143211515 disk.rbd1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.355006e+06 disk.sda3.out‚232.3125 disk.dm-0.in‚ 4.9296875 disk.nvme0n1.in‚ 4.9296875 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚ 4.9296875 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 232.3515625$ memory_percent‚98.93651008605957' proc.memory.availableMB‚ 174.2421875* proc.memory.percent‚17.399215698242188 disk.rbd1.out‚ 1.90283203125& disk.nvme0n1.out‚16400.94091796875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚16400.94091796875 cpu‚25.28552355552459% disk./.usageGB‚1098.3750534057617 disk.rbd4.in‚0 disk.rbd4.out‚0 proc.memory.rssMB‚ 2850.68757Ð÷i2 - _wandbruntime‚3758  _runtime‚3758îÒ7Ç¡Á % _timestamp‚1.7884202873043315e+09 _runtime‚3758.054273008 _step‚317 & lr-AdamW/pg1‚0.0006872884413195124 & lr-AdamW/pg2‚0.0006872884413195124 trainer/global_step‚3899TŒPÌ¢Æ trainer/global_step‚3899 % _timestamp‚1.7884202873043315e+09 _runtime‚3758.054273008 _step‚317 & lr-AdamW/pg1‚0.0006872884413195124 & lr-AdamW/pg2‚0.0006872884413195124½C§n#2£- _wandbruntime‚3758  _runtime‚3758,„ ¤ÙO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68281567 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26190949 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚146271113526272 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101468188 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚161717199508992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173407345 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71939157 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚153491393570918 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101468188 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153491634259426 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7371207368704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 % _timestamp‚1.7884202873050768e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71939157 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161716953302528 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154475638167552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165757765588992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42089966 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7370938547200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161717199508992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7241543338496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68280915 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154475656170496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _step‚318*zä'¥ÞO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101468188 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153491634259426 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7371207368704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 % _timestamp‚1.7884202873050768e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71939157 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161716953302528 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154475638167552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165757765588992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42089966 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7370938547200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161717199508992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7241543338496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68280915 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154475656170496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _step‚318 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68281567 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26190949 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚146271113526272 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101468188 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚161717199508992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173407345 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71939157 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚153491393570918¾ß>Åü2¦- _wandbruntime‚3758  _runtime‚3758„™Êé&§ãM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7243152829440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68298749 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173450615 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68298100 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚3758.918575889 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42100321 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7243381046272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71956878 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚154513976111616 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173476313 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165798935285248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165798911345152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7222129483606 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101493739 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154513994114560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7243381046272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101493739 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42100323 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7243152829440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 trainer/global_step‚3899 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71956876 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7372803057152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154513976111616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26198426 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173172013442048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173476313 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 _step‚319 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7373078156800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161757375160832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101519435 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26197779 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173171714402304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153529769117090 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161757128941056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884202881679938e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173450615•$ƒ¨èM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71956876 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7372803057152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154513976111616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26198426 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173172013442048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173476313 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 _step‚319 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7373078156800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161757375160832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101519435 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26197779 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173171714402304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153529769117090 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161757128941056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884202881679938e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173450615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7243152829440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68298749 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173450615 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68298100 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚3758.918575889 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42100321 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7243381046272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71956878 OADeviceStatsMonitor.on_train_batª¶Ñqkch_end/requested_bytes.all.current‚ 240688716 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚154513976111616 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173476313 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165798935285248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165798911345152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7222129483606 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101493739 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154513994114560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7243381046272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101493739 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42100323 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7243152829440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 trainer/global_step‚3899 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F:iß'¶ÙO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚175280821649408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42607718 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚163725749642752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102749090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3803.124463562 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚163725749642752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175574197 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72825109 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72825107 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102749090 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7312154304362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7464571247616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72825109 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102774760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚148086236419072 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7333214272000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148086219379712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 trainer/global_step‚3949 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102774760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7333442475520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚163725995849216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884203323742504e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26515352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚167816226461696 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156392553373696 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175599869 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚163725995849216 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7333442475520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156392535370752 _step‚322 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69123070 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42607716ôGz=ä'·ÞO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚175280526790656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156392535370752 _step‚322 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69123070 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42607716 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚175280821649408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42607718 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚163725749642752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102749090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3803.124463562 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚163725749642752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175574197 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72825109 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72825107 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102749090 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7312154304362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7464571247616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72825109 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102774760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚148086236419072 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7333214272000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148086219379712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 trainer/global_step‚3949 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102774760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7333442475520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚163725995849216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884203323742504e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26515352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚167816226461696 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156392553373696 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175599869 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚163725995849216 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7333442475520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69138981 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7335280183296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156430873314816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72842828 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚175323859231744 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148122538876928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69139625 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102774641 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884203332311134e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7466439133696 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚163766171501056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚167857396157952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156430891317760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7335280183296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚148122521837568 trainer/global_step‚3949 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72842826 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚163766171501056 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156430873314816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7313986704170 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7335051966464 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚163765925281280 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42618073 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156430891317760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚175323570677760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175617467 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175617467 _step‚323 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 264613888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7313763054814 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175643165 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26520910 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102800337 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26521552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155436284892382 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175643165 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175643165 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69138981 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7335280183296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156430873314816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72842828 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚175323859231744 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148122538876928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69139625 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102774641 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884203332311134e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7466439133696 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚163766171501056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚167857396157952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156430891317760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7335280183296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚148122521837568 trainer/global_step‚3949 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72842826 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚163766171501056 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156430873314816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7313986704170 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7335051966464 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚163765925281280 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42618073 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156430891317760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚175323570677760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175617467 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175617467 _step‚323 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 264613888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7313763054814 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175643165 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26520910 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102800337 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26521552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155436284892382 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ÃÍx˜2»- _wandbruntime‚3803  _runtime‚3803d)U¼Ï _step‚324 0loss/FBt1/assignment_loss‚0.7016746457652516 .loss/FRt2/detection_loss‚1.555469714552164 /loss/FRt4/detection_loss‚2.5822110209200115 /loss/FRt4/assignment_loss‚4.286455955770281 0loss/FBt2/assignment_loss‚0.1628979459814928 0loss/FRt3/assignment_loss‚3.8118350505828857 /loss/FRt1/assignment_loss‚3.316209374590123 2loss/SRqqt2/assignment_loss‚1.2321107922920158 1loss/SRqqt2/detection_loss‚0.8441614947148732 -loss/FBt1/detection_loss‚2.42268950621096 .loss/FBt4/detection_loss‚2.892737165093422 epoch‚3 0loss/FRt2/assignment_loss‚1.9364363050460816 /loss/FRt1/detection_loss‚2.4274241294949612 /loss/FRt3/detection_loss‚2.3939591738912793 /loss/SRqqt1/detection_loss‚2.36304906332815 1loss/SRqqt3/detection_loss‚1.9193206336700692 _runtime‚3803.983036113 /loss/FBt4/assignment_loss‚0.823307258047862 2loss/SRqqt1/assignment_loss‚2.0499297509322294 1loss/FBt3/assignment_loss‚0.32657733585801907 trainer/global_step‚3949 % _timestamp‚1.7884203332332168e+09 /loss/FBt3/detection_loss‚2.9646337032318115 %loss/total_loss‚47.70900837829429 2loss/SRqqt3/assignment_loss‚1.5607113956280474 1loss/SRqqt4/assignment_loss‚2.142621566851934 0loss/SRqqt4/detection_loss‚2.250503204571895 /loss/FBt2/detection_loss‚0.7420821912684723¸ Z½Ô 2loss/SRqqt1/assignment_loss‚2.0499297509322294 1loss/FBt3/assignment_loss‚0.32657733585801907 trainer/global_step‚3949 % _timestamp‚1.7884203332332168e+09 /loss/FBt3/detection_loss‚2.9646337032318115 %loss/total_loss‚47.70900837829429 2loss/SRqqt3/assignment_loss‚1.5607113956280474 1loss/SRqqt4/assignment_loss‚2.142621566851934 0loss/SRqqt4/detection_loss‚2.250503204571895 /loss/FBt2/detection_loss‚0.7420821912684723 _step‚324 0loss/FBt1/assignment_loss‚0.7016746457652516 .loss/FRt2/detection_loss‚1.555469714552164 /loss/FRt4/detection_loss‚2.5822110209200115 /loss/FRt4/assignment_loss‚4.286455955770281 0loss/FBt2/assignment_loss‚0.1628979459814928 0loss/FRt3/assignment_loss‚3.8118350505828857 /loss/FRt1/assignment_loss‚3.316209374590123 2loss/SRqqt2/assignment_loss‚1.2321107922920158 1loss/SRqqt2/detection_loss‚0.8441614947148732 -loss/FBt1/detection_loss‚2.42268950621096 .loss/FBt4/detection_loss‚2.892737165093422 epoch‚3 0loss/FRt2/assignment_loss‚1.9364363050460816 /loss/FRt1/detection_loss‚2.4274241294949612 /loss/FRt3/detection_loss‚2.3939591738912793 /loss/SRqqt1/detection_loss‚2.36304906332815 1loss/SRqqt3/detection_loss‚1.9193206336700692 _runtime‚3803.983036113 /loss/FBt4/assignment_loss‚0.823307258047862Ä@ÁHJ›¾‚(: óÁäÔ§½¿x gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.611' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.322200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•zuHq¿‚(:æ óÁäÔèœë{ disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.dm-0.out‚16600.48828125* proc.memory.percent‚17.399215698242188 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 234.81640625! disk.rbd2.out‚143.18212890625 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 4.9765625 disk.sda2.in‚0$ memory_percent‚98.93817901611328 network.sent‚ 8.827799e+06 disk.rbd0.out‚0% disk./.usageGB‚1098.3759078979492% disk.nvme0c0n1.out‚16600.48828125 disk.sda.in‚ 7.36328125 disk.sda.out‚ 234.85546875# disk.nvme0n1.out‚16600.48828125) disk./.usagePercent‚73.71292877780283 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.rbd1.out‚ 1.90283203125 disk.nvme0n1.in‚ 4.9765625 proc.memory.rssMB‚ 2850.6875 network.recv‚ 1.387732e+06 disk.rbd3.out‚0 disk.sda3.out‚ 234.81640625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0% proc.memory.availableMB‚ 173.96875 cpu‚25.28170690487551à¬hŠÀ‚(:… ‚ÂäÔÙБx gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ylP™uÁ‚(:ê ‚ÂäÔ‘ôÔz disk.dm-1.in‚7.3125 proc.memory.rssMB‚ 2850.6875 disk.sda3.out‚ 235.3359375 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚ 4.9765625 disk.sda.in‚ 7.36328125 disk.dm-1.out‚ 235.3359375 proc.cpu.threads‚17 disk.sda.out‚235.375 disk.rbd1.in‚0 disk.rbd1.out‚ 1.90283203125 disk.dm-0.in‚ 4.9765625# disk.dm-0.out‚16699.95751953125* proc.memory.percent‚17.399215698242188 disk.drbd1777.out‚0 disk.sda2.out‚ 0.0390625 network.sent‚ 8.936196e+06% disk./.usageGB‚1098.3759078979492 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0$ memory_percent‚98.93898963928223& disk.nvme0n1.out‚16699.95751953125 network.recv‚ 1.393845e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.sda1.out‚0! disk.rbd2.out‚143.18212890625 cpu‚25.28075558345059( disk.nvme0c0n1.out‚16699.95751953125 disk.rbd3.out‚0 disk.sda3.in‚7.3125' proc.memory.availableMB‚ 173.8359375) disk./.usagePercent‚73.71292877780283 disk.rbd4.out‚0 disk.nvme0n1.in‚ 4.9765625Ê3˜››Â‚(: ‘ÂäÔð©³x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.969' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.593799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(3Û?uÂ(:ê ‘ÂäÔ§¢Ëz disk.rbd0.out‚0 disk.sda.in‚ 7.36328125# disk.dm-0.out‚16798.99267578125# proc.memory.rssMB‚ 2850.69140625 network.sent‚ 8.942393e+06) disk./.usagePercent‚73.71292877780283 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda3.out‚235.75 disk.rbd1.out‚ 1.90283203125 disk.sda1.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚16798.99267578125( proc.memory.availableMB‚ 173.97265625* proc.memory.percent‚17.399239540100098 network.recv‚ 1.398756e+06% disk./.usageGB‚1098.3759078979492 disk.nvme0n1.in‚ 4.9765625 disk.sda3.in‚7.3125 disk.dm-1.in‚7.3125 disk.dm-1.out‚235.75 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.dm-0.in‚ 4.9765625 cpu‚25.279861375314585 disk.rbd4.out‚0( disk.nvme0c0n1.out‚16798.99267578125 disk.sda.out‚ 235.7890625 disk.sda2.in‚0 disk.rbd3.in‚0! disk.rbd2.out‚143.26025390625 disk.rbd4.in‚0$ memory_percent‚98.93815517425537 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625–Na2Ä- _wandbruntime‚3848  _runtime‚3848¿JtŒÆÅÀ & lr-AdamW/pg2‚0.0006790521768870961 trainer/global_step‚3999 $ _timestamp‚1.788420377488361e+09 _runtime‚3848.238256856 _step‚325 & lr-AdamW/pg1‚0.0006790521768870961ßýËÆÅ _step‚325 & lr-AdamW/pg1‚0.0006790521768870961 & lr-AdamW/pg2‚0.0006790521768870961 trainer/global_step‚3999 $ _timestamp‚1.788420377488361e+09 _runtime‚3848.238256856Åâ9øÇ2Ç- _wandbruntime‚3848  _runtime‚3848²ÈãúÈÙO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚165734545982976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169874711274496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177741049 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157305147187442 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚165734545982976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158309450576896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7425113409024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _runtime‚3848.239203549 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177766721 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73711057 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43125466 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104029992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158309432573952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73711059 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169874735214592 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177432632446976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7403787875570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KEDeviceStatsMonitor.onÍLÿ å_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 % _timestamp‚1.7884203774890516e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7557897232384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7425341612544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚165734792189440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158309450576896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7557622119424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149901359311872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149901342272512 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104055662 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69961861 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 trainer/global_step‚3999 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7403787875570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 % _timestamp‚1.7884203774890516e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7557897232384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7425341612544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚165734792189440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158309450576896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7557622119424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149901359311872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149901342272512 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104055662 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69961861 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 trainer/global_step‚3999 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚165734545982976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169874711274496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177741049 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157305147187442 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚165734545982976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158309450576896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7425113409024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _runtime‚3848.239203549 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177766721 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616Æ°ÆÔ«2Ê- _wandbruntime‚3849  _runtime‚3849G.«„é&ËãM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158347770518016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _runtime‚3849.100064639 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7427179320320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158347788520960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177475665820672 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158347788520960 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177475362586624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7559481615872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177810017 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43135823 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚165774721621504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26842578 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26841920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177810017 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177784319 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104081239 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43135821 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69978401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149937661769728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104081239 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚169915904910848 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7426951103488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7559760909824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73728778 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 % _timestamp‚1.7884203783494825e+09 _step‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚165774967841280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177784319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69977741 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0/ùÌèM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104081239 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43135821 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69978401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149937661769728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104081239 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚169915904910848 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7426951103488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7559760909824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73728778 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 % _timestamp‚1.7884203783494825e+09 _step‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚165774967841280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177784319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69977741 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158347770518016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _runtime‚3849.100064639 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7427179320320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158347788520960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177475665820672 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158347788520960 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177475362586624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7559481615872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177810017 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43135823 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚165774721621504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26842578 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26841920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚Œ»ªùõ 561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177810017 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177784319 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7517240749568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚681 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27160479 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 179933573 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43643216 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43643218 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179584709071872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171933196087296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151716482204672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 trainer/global_step‚4049 _step‚330 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105310894 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚160226347780096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27161158 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167743342323200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7495197797630 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167743588529664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262P’œä'ÛÞO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43643218 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179584709071872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171933196087296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151716482204672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 trainer/global_step‚4049 _step‚330 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105310894 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚160226347780096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27161158 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167743342323200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7495197797630 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167743588529664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160226347780096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7517240749568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚681 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27160479 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 179933573 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43643216 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243Ê+~Æs2Ü- _wandbruntime‚3894  _runtime‚3894ø¼(“ÝãM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70820219 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚4049 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179951171 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74614728 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27166648 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freedF®1`V!‚167783517961728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43653573 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚171974365783552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7653085650432 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 % _timestamp‚1.7884204234691725e+09 _runtime‚3894.219785745 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7518850240512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105362141 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70820866 _step‚331 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74614726 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚160264685724160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160264685724160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151752784662528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167783764181504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7519078457344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179951171 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74614728 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160264667721216 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167783764181504 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚159249797820398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7519078457344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74614726 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179976869 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179627451433984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105336445 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7497253846586 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160264667721216 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151752767623168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚171974389723648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43653571 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944f¶1ýî&ÞèM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160264667721216 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151752767623168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚171974389723648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43653571 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70820219 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚4049 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179951171 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74614728 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27166648 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167783517961728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43653573 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚171974365783552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7653085650432 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 % _timestamp‚1.7884204234691725e+09 _runtime‚3894.219785745 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7518850240512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105362141 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70820866 _step‚331 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74614726 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚160264685724160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160264685724160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151752784662528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167783764181504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7519078457344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179951171 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74614728 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160264667721216 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167783764181504 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚159249797820398 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7519078457344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74614726 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179976869 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179627451433984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105336445 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7497253846586 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624Ë_ _92ß- _wandbruntime‚3894  _runtime‚3894_p5[àÕ /loss/FRt4/assignment_loss‚3.515940503376286 0loss/FBt1/assignment_loss‚0.7337769004474239 epoch‚3 _step‚332 .loss/FRt3/detection_loss‚2.580459446460009 1loss/SRqqt1/detection_loss‚2.4535385852350906 1loss/SRqqt4/detection_loss‚1.8794767546157043 1loss/FBt2/assignment_loss‚0.16131597441141296 trainer/global_step‚4049 2loss/SRqqt4/assignment_loss‚1.9285591738298535 /loss/FRt2/detection_loss‚2.1769398430155382 1loss/SRqqt3/detection_loss‚1.8177263708068774 /loss/FBt2/detection_loss‚0.5716431147361897 .loss/FRt1/detection_loss‚2.605817144879928 /loss/FRt4/detection_loss‚2.5375666865488378 1loss/SRqqt2/detection_loss‚1.3805020240220156 /loss/FBt1/detection_loss‚2.7046418264508247 .loss/FBt3/detection_loss‚3.736633598804474 2loss/SRqqt2/assignment_loss‚1.2816156103529714 1loss/FBt3/assignment_loss‚0.10857765655964613 .loss/FBt4/detection_loss‚2.637590902192252 /loss/FRt1/assignment_loss‚4.312029205835783 0loss/FRt2/assignment_loss‚1.8592745380269156 1loss/SRqqt3/assignment_loss‚2.463467442358916 1loss/FBt4/assignment_loss‚0.34016449230590035 _runtime‚3894.220964777 0loss/FRt3/assignment_loss‚3.6158785104751585 2loss/SRqqt1/assignment_loss‚2.0262963359846786 %loss/total_loss‚49.42943264173269 % _timestamp‚1.7884204234711313e+09Ù1Ô`áÚ 2loss/SRqqt1/assignment_loss‚2.0262963359846786 %loss/total_loss‚49.42943264173269 % _timestamp‚1.7884204234711313e+09 /loss/FRt4/assignment_loss‚3.515940503376286 0loss/FBt1/assignment_loss‚0.7337769004474239 epoch‚3 _step‚332 .loss/FRt3/detection_loss‚2.580459446460009 1loss/SRqqt1/detection_loss‚2.4535385852350906 1loss/SRqqt4/detection_loss‚1.8794767546157043 1loss/FBt2/assignment_loss‚0.16131597441141296 trainer/global_step‚4049 2loss/SRqqt4/assignment_loss‚1.9285591738298535 /loss/FRt2/detection_loss‚2.1769398430155382 1loss/SRqqt3/detection_loss‚1.8177263708068774 /loss/FBt2/detection_loss‚0.5716431147361897 .loss/FRt1/detection_loss‚2.605817144879928 /loss/FRt4/detection_loss‚2.5375666865488378 1loss/SRqqt2/detection_loss‚1.3805020240220156 /loss/FBt1/detection_loss‚2.7046418264508247 .loss/FBt3/detection_loss‚3.736633598804474 2loss/SRqqt2/assignment_loss‚1.2816156103529714 1loss/FBt3/assignment_loss‚0.10857765655964613 .loss/FBt4/detection_loss‚2.637590902192252 /loss/FRt1/assignment_loss‚4.312029205835783 0loss/FRt2/assignment_loss‚1.8592745380269156 1loss/SRqqt3/assignment_loss‚2.463467442358916 1loss/FBt4/assignment_loss‚0.34016449230590035 _runtime‚3894.220964777 0loss/FRt3/assignment_loss‚3.6158785104751585ÌY\þþŽâ‚(:ƒ ÍÂäÔãý x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H;µyã‚(:î ÍÂäÔÔ£½z$ disk./.usageGB‚1098.375877380371# disk.dm-0.out‚17097.12646484375 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.sda3.out‚ 238.1015625' proc.memory.availableMB‚ 173.4921875 disk.rbd3.in‚0 proc.cpu.threads‚17 network.sent‚ 9.0684e+06 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 1.90283203125 disk.sda.out‚ 238.140625 network.recv‚ 1.417276e+06 disk.dm-0.in‚ 4.9765625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125& disk.nvme0n1.out‚17097.12646484375 disk.rbd2.out‚145.0263671875 disk.sda.in‚ 7.36328125# proc.memory.rssMB‚ 2850.69140625* proc.memory.percent‚17.399239540100098 disk.nvme0n1.in‚ 4.9765625 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 238.1015625( disk.nvme0c0n1.out‚17097.12646484375 disk.rbd1.in‚0 cpu‚25.276286639342352 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.out‚0 disk.sda3.in‚7.3125) disk./.usagePercent‚73.71292672974256 disk.rbd4.in‚0 disk.rbd3.out‚0$ memory_percent‚98.94108772277832³ÌÖJ›ä‚(: ÜÂäÔ÷·¥x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.799' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.159800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»|—å‚(:ô ÜÂäÔº•–z disk.nvme0c0n1.in‚ 4.9765625# proc.memory.rssMB‚ 2850.69140625 cpu‚25.27536616020452 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 1.90283203125 disk.sda2.in‚0 disk.rbd4.out‚0& disk.nvme0n1.out‚17199.79541015625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚17199.79541015625 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 238.66796875$ memory_percent‚98.94282817840576 network.sent‚ 9.176272e+06 disk.nvme0n1.in‚ 4.9765625 disk.sda2.out‚ 0.0390625 disk.sda3.in‚7.3125$ disk./.usageGB‚1098.378993988037# disk.dm-0.out‚17199.79541015625 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 238.62890625 disk.drbd1777.in‚ 1.03515625 disk.sda3.out‚ 238.62890625* proc.memory.percent‚17.399239540100098) disk./.usagePercent‚73.71313588789803 disk.rbd3.out‚0 network.recv‚ 1.422605e+06 disk.rbd2.in‚0 disk.rbd2.out‚145.0380859375 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125( proc.memory.availableMB‚ 173.20703125 disk.dm-0.in‚ 4.9765625 disk.rbd0.out‚0! î›æ‚(: ëÂäÔ÷˜Ãx gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.526' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.705199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆK<ƒwç‚(:ì ëÂäÔþêñz( proc.memory.availableMB‚ 173.20703125 proc.cpu.threads‚17 disk.rbd4.out‚0 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.sda2.in‚0 disk.nvme0n1.in‚ 4.9765625 disk.sda2.out‚ 0.0390625% disk./.usageGB‚1098.3790168762207 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.90283203125 cpu‚25.27450217783102) disk./.usagePercent‚73.71313742394324( disk.nvme0c0n1.out‚17200.06494140625# disk.dm-0.out‚17200.06494140625 disk.dm-1.in‚7.3125 network.sent‚ 9.182294e+06 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd3.in‚0 disk.sda.out‚ 239.1015625& disk.nvme0n1.out‚17200.06494140625 disk.dm-1.out‚239.0625$ memory_percent‚98.94282817840576 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.sda3.out‚239.0625 disk.dm-0.in‚ 4.9765625* proc.memory.percent‚17.399239540100098 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚145.1005859375# proc.memory.rssMB‚ 2850.69140625 network.recv‚ 1.427023e+06Ô57¦2è- _wandbruntime‚3938  _runtime‚3938‚‘½ ÇéÁ trainer/global_step‚4099 % _timestamp‚1.7884204680628633e+09 _runtime‚3938.812779227 _step‚333 & lr-AdamW/pg1‚0.0006705842747170542 & lr-AdamW/pg2‚0.0006705842747170542¦Ñü!ÌêÆ & lr-AdamW/pg2‚0.0006705842747170542 trainer/global_step‚4099 % _timestamp‚1.7884204680628633e+09 _runtime‚3938.812779227 _step‚333 & lr-AdamW/pg1‚0.0006705842747170542Í B®ì2ë- _wandbruntime‚3938  _runtime‚3938öX6å5ìÙO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27486566 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75482957 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106617466 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106591796 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7609139886592 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71647532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 _step‚334 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚169752384869888 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181736675229696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 260432896 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚161118419426950 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162143226980352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7608911683072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7608911683072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 284372992 trainer/global_step‚4099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75482959 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75482957 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7587055017986 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162143244983296 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106617466 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚169752138663424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75482959 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7586831368838 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_tr2Pªain_batch_start/allocated_bytes.small_pool.peak‚ 541892608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153531605097472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7609139886592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7744709956608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153531588058112 _runtime‚3938.813663022 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169752138663424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106591796 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 % _timestamp‚1.7884204680635805e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚173991704840192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚173991680900096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚161118660115458 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚169752384869888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27487224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182100425 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181736390856704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182074753 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162143226980352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162143244983296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44160966í ­%ä'íÞO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚161118419426950 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162143226980352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7608911683072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7608911683072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 284372992 trainer/global_step‚4099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75482959 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75482957 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7587055017986 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162143244983296 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106617466 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚169752138663424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75482959 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7586831368838 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153531605097472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7609139886592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7744709956608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153531588058112 _runtime‚3938.813663022 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169752138663424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106591796 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 % _timestamp‚1.7884204680635805e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚173991704840192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚173991680900096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚161118660115458 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚169752384869888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27487224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182100425 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181736390856704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182074753 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162143226980352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162143244983296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44160966 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27486566 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75482957 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106617466 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106591796 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7609139886592 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71647532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 _step‚334 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚169752384869888 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181736675229696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 260432896 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299ÎìÐe!2î- _wandbruntime‚3939  _runtime‚3939môòé&ïãM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7610977594368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106643043 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚161156794973122 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚161156554284406 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7610749377536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174032874536448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7746575400448 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162181564924416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7610749377536 trainer/global_step‚4099 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27492846 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7588887417794 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106617347 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75500678 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚181779720842240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75500676 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182143721 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71664167 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7610977594368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153567890515968 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚169792560521728 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182143721 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75500678 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7588663768438 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182118023 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181779425996800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75500676 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 % _timestamp‚1.7884204689123995e+09=¸ª 4!ðèM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7588663768438 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182118023 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181779425996800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75500676 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 % _timestamp‚1.7884204689123995e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7610977594368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106643043 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚161156794973122 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚161156554284406 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7610749377536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174032874536448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7746575400448 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162181564924416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7610749377536 trainer/global_step‚4099 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27492846 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7588887417794 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106617347 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75500678 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚181779720842240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75500676 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182143721 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71664167 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7610977594368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153567890515968 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚169792560521728 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182143721 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75500678 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128Ït W2ñ- _wandbruntime‚3939  _runtime‚3939’àVòÐ 0loss/SRqqt2/detection_loss‚0.850812048397281 .loss/FBt4/detection_loss‚2.765922921044486 1loss/FBt4/assignment_loss‚0.39849445170589853 /loss/FRt1/assignment_loss‚4.333962779978047 1loss/SRqqt3/assignment_loss‚1.871852602213621 0loss/FBt1/assignment_loss‚0.6854014313880472 1loss/FBt3/assignment_loss‚0.05288086545382006 1loss/SRqqt3/detection_loss‚1.8779141920804978 0loss/SRqqt4/detection_loss‚2.078752095169491 /loss/FRt3/assignment_loss‚4.762374209506171 % _timestamp‚1.7884204689139872e+09 -loss/FRt1/detection_loss‚2.22992736959587 /loss/FRt4/detection_loss‚2.3459536893801256 /loss/FBt2/detection_loss‚0.6602590780138321 trainer/global_step‚4099 /loss/FBt1/detection_loss‚2.6288708448410034 /loss/FBt3/detection_loss‚3.6670532723267875 epoch‚3 0loss/FRt2/assignment_loss‚1.5336591866281297 0loss/FRt4/assignment_loss‚3.6121100584665933 1loss/FBt2/assignment_loss‚0.23406944565330828 _runtime‚3939.663775609 -loss/FRt2/detection_loss‚1.74337258256144 .loss/FRt3/detection_loss‚2.664272820665723 &loss/total_loss‚48.345412733831274 2loss/SRqqt2/assignment_loss‚1.2371596997589662 _step‚336 1loss/SRqqt1/assignment_loss‚1.925020211296422 1loss/SRqqt4/assignment_loss‚2.018271040916443 1loss/SRqqt1/detection_loss‚2.1670458367892675é;o³[óÕ &loss/total_loss‚48.345412733831274 2loss/SRqqt2/assignment_loss‚1.2371596997589662 _step‚336 1loss/SRqqt1/assignment_loss‚1.925020211296422 1loss/SRqqt4/assignment_loss‚2.018271040916443 1loss/SRqqt1/detection_loss‚2.1670458367892675 0loss/SRqqt2/detection_loss‚0.850812048397281 .loss/FBt4/detection_loss‚2.765922921044486 1loss/FBt4/assignment_loss‚0.39849445170589853 /loss/FRt1/assignment_loss‚4.333962779978047 1loss/SRqqt3/assignment_loss‚1.871852602213621 0loss/FBt1/assignment_loss‚0.6854014313880472 1loss/FBt3/assignment_loss‚0.05288086545382006 1loss/SRqqt3/detection_loss‚1.8779141920804978 0loss/SRqqt4/detection_loss‚2.078752095169491 /loss/FRt3/assignment_loss‚4.762374209506171 % _timestamp‚1.7884204689139872e+09 -loss/FRt1/detection_loss‚2.22992736959587 /loss/FRt4/detection_loss‚2.3459536893801256 /loss/FBt2/detection_loss‚0.6602590780138321 trainer/global_step‚4099 /loss/FBt1/detection_loss‚2.6288708448410034 /loss/FBt3/detection_loss‚3.6670532723267875 epoch‚3 0loss/FRt2/assignment_loss‚1.5336591866281297 0loss/FRt4/assignment_loss‚3.6121100584665933 1loss/FBt2/assignment_loss‚0.23406944565330828 _runtime‚3939.663775609 -loss/FRt2/detection_loss‚1.74337258256144 .loss/FRt3/detection_loss‚2.664272820665723ÐS*ôô‚(:„ úÂäÔõàÁx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.392' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@NË õ‚(:ô úÂäÔóþ™{ disk.dm-0.in‚ 4.9765625 disk.dm-1.in‚7.3125% disk./.usageGB‚1098.3790435791016 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd0.in‚0 cpu‚25.273650029519832 disk.sda.out‚ 239.66015625 disk.rbd1.out‚ 1.90283203125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚146.6279296875 proc.cpu.threads‚17 disk.sda.in‚ 7.36328125 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd1.in‚0$ memory_percent‚98.94113540649414* proc.memory.percent‚17.399239540100098 disk.rbd3.out‚0 network.sent‚ 9.188183e+06 network.recv‚ 1.431021e+06 disk.rbd4.in‚0& disk.nvme0n1.out‚17300.25244140625# disk.dm-0.out‚17300.25244140625 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 239.62109375 disk.sda1.out‚0 disk.sda3.out‚ 239.62109375 disk.nvme0n1.in‚ 4.9765625 disk.rbd0.out‚0& proc.memory.availableMB‚ 173.484375# proc.memory.rssMB‚ 2850.69140625 disk.rbd4.out‚0( disk.nvme0c0n1.out‚17300.25244140625) disk./.usagePercent‚73.71313921599597 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.sda3.in‚7.3125Þƒ¥âö‚(:… ‰ÃäÔ¨Á”x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.896' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ«d|÷‚(:ñ ‰ÃäÔ‹¦¹z% disk./.usageGB‚1098.3782501220703 disk.drbd1777.out‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 240.109375& disk.nvme0n1.out‚17399.31787109375 disk.sda.out‚ 240.1484375 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 4.9765625) disk./.usagePercent‚73.71308596642885 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.dm-0.in‚ 4.9765625 cpu‚25.272818519419758 proc.cpu.threads‚17 disk.rbd2.out‚146.7685546875 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0# disk.dm-0.out‚17399.31787109375 disk.dm-1.in‚7.3125* proc.memory.percent‚17.399239540100098 disk.rbd4.in‚0 disk.sda.in‚ 7.36328125( proc.memory.availableMB‚ 173.03515625 disk.nvme0n1.in‚ 4.9765625( disk.nvme0c0n1.out‚17399.31787109375 network.sent‚ 9.296971e+06 network.recv‚ 1.437761e+06 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 240.109375 disk.rbd4.out‚0 disk.rbd1.out‚ 1.90283203125$ memory_percent‚98.94387722015381# proc.memory.rssMB‚ 2850.69140625òµøŽø‚(:ƒ ˜ÃäÔ¸£ˆx gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.768' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýKÿiù‚(:Þ ˜ÃäÔçϱz disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0* proc.memory.percent‚17.399239540100098 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 240.6640625 disk.rbd2.in‚0% proc.memory.availableMB‚ 173.03125 disk.nvme0c0n1.in‚ 4.9765625% disk.nvme0c0n1.out‚17498.67578125# disk.nvme0n1.out‚17498.67578125 disk.dm-0.out‚17498.67578125 disk.rbd2.out‚146.7841796875 network.sent‚ 9.302356e+06) disk./.usagePercent‚73.71309057456446 disk.rbd4.out‚0 disk.sda3.out‚240.625 disk.nvme0n1.in‚ 4.9765625 disk.sda2.in‚0# proc.memory.rssMB‚ 2850.69140625 cpu‚25.271984086903867 network.recv‚ 1.441392e+06 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.sda3.in‚7.3125 disk.rbd1.in‚0 disk.dm-1.in‚7.3125 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.dm-0.in‚ 4.9765625 disk.dm-1.out‚240.625$ memory_percent‚98.94390106201172$ disk./.usageGB‚1098.378318786621 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 1.90283203125P’Ñœ2ú- _wandbruntime‚3984  _runtime‚3984êt|ÇûÁ _step‚337 & lr-AdamW/pg1‚0.0006662656578556919 & lr-AdamW/pg2‚0.0006662656578556919 trainer/global_step‚4149 % _timestamp‚1.7884205132738464e+09 _runtime‚3984.023800308ŽƒÌüÆ _runtime‚3984.023800308 _step‚337 & lr-AdamW/pg1‚0.0006662656578556919 & lr-AdamW/pg2‚0.0006662656578556919 trainer/global_step‚4149 % _timestamp‚1.7884205132738464e+09Ñ«S»†2ý- _wandbruntime‚3984  _runtime‚3984´[¿íÝ'þ×O GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155346727990272 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155346710950912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7838203298816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72492827 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚171761181210112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7700810820096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27814789 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7701039023616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚171761181210112 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163025416579466 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107872698 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27814111 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚164060142186496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44678718 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107898368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44678716 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚338 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7678688589194 _runtime‚3984.024732093 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚164060142186496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚164060124183552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107898368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7700810820096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184267277 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76368907 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76368907 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171760935003648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚171760935003648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163025175890958 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7701039023616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184241605 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107872698 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76368909 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72493507 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚176050189652992 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176050165712896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76368909 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7678464940046 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164060124183552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 trainer/global_step‚4149 # _timestamp‚1.78842051327462e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397W0EBâ'ÿÜO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 trainer/global_step‚4149 # _timestamp‚1.78842051327462e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155346727990272 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155346710950912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7838203298816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72492827 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚171761181210112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7700810820096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27814789 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7701039023616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚171761181210112 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163025416579466 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107872698 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27814111 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚164060142186496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44678718 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107898368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44678716 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚338 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7678688589194 _runtime‚3984.024732093 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚164060142186496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚164060124183552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107898368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7700810820096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184267277 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76368907 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76368907 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171760935003648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚171760935003648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163025175890958 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7701039023616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184241605 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107872698 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76368909 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72493507 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚176050189652992 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176050165712896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76368909 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7678464940046 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164060124183552Ò¼‹øÃ2€- _wandbruntime‚3984  _runtime‚3984\ƒ»mì ãM SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107923945 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163063310748414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7840342318592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7680297339646 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76386628 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72509498 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XFDeviceStatsMonitor.on_train_batch_end/allocated_byteø»³¡ýs.small_pool.freed‚ 7702648514560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27821056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163063551437130 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184284875 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76386626 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164098480130560 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44689073 _runtime‚3984.882264023 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183931701667840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107923945 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27820427 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183931406822400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176091335409152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884205141315513e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107898249 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72510129 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7840071413248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107898249 _step‚339 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7702876731392 trainer/global_step‚4149 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚164098462127616 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76386626 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚171801110642176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184310573 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184284875 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7702648514560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164098462127616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696‚3;‚î&‚èM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76386628 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72509498 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7702648514560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27821056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163063551437130 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184284875 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76386626 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164098480130560 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44689073 _runtime‚3984.882264023 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183931701667840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107923945 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27820427 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183931406822400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176091335409152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884205141315513e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107898249 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72510129 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7840071413248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107898249 _step‚339 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7702876731392 trainer/global_step‚4149 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚164098462127616 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76386626 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚171801110642176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184310573 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184284875 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7702648514560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164098462127616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107923945 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163063310748414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7840342318592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7680297339646 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÓÈüa‰2ƒ- _wandbruntime‚3984  _runtime‚3984#ê•Z„Ô /loss/FRt4/detection_loss‚2.4176151156425476 2loss/SRqqt4/assignment_loss‚1.9561614818100272 .loss/FRt2/detection_loss‚1.721583002143436 2loss/SRqqt2/assignment_loss‚1.3576895425561815 /loss/FRt3/detection_loss‚2.8178977981591835 .loss/FBt3/detection_loss‚4.462648431460063 _runtime‚3984.883487368 .loss/FBt1/detection_loss‚1.898771244287491 /loss/FBt2/detection_loss‚0.6030381516265195 2loss/FBt3/assignment_loss‚0.049064720301733665 /loss/FRt1/detection_loss‚2.6684620762243867 /loss/SRqqt1/detection_loss‚2.74575221997041 1loss/SRqqt4/detection_loss‚1.8378805523288662 /loss/FRt1/assignment_loss‚4.506610654294491 0loss/FRt2/assignment_loss‚1.6694335164847198 1loss/SRqqt2/detection_loss‚1.0999198533594607 /loss/FBt4/detection_loss‚2.7395259618759153 $loss/total_loss‚48.2613696165281 _step‚340 0loss/FRt4/assignment_loss‚3.5062023793778767 2loss/SRqqt1/assignment_loss‚2.1342526404903484 /loss/FRt3/assignment_loss‚3.707302283018063 0loss/FBt1/assignment_loss‚0.3496567405760288 0loss/SRqqt3/detection_loss‚1.952842744688193 trainer/global_step‚4149 % _timestamp‚1.7884205141336653e+09 epoch‚3 2loss/SRqqt3/assignment_loss‚1.8664935862955947 1loss/FBt2/assignment_loss‚0.14938565098035841 1loss/FBt4/assignment_loss‚0.04317926857620478Î1J_…Ù /loss/FRt3/assignment_loss‚3.707302283018063 0loss/FBt1/assignment_loss‚0.3496567405760288 0loss/SRqqt3/detection_loss‚1.952842744688193 trainer/global_step‚4149 % _timestamp‚1.7884205141336653e+09 epoch‚3 2loss/SRqqt3/assignment_loss‚1.8664935862955947 1loss/FBt2/assignment_loss‚0.14938565098035841 1loss/FBt4/assignment_loss‚0.04317926857620478 /loss/FRt4/detection_loss‚2.4176151156425476 2loss/SRqqt4/assignment_loss‚1.9561614818100272 .loss/FRt2/detection_loss‚1.721583002143436 2loss/SRqqt2/assignment_loss‚1.3576895425561815 /loss/FRt3/detection_loss‚2.8178977981591835 .loss/FBt3/detection_loss‚4.462648431460063 _runtime‚3984.883487368 .loss/FBt1/detection_loss‚1.898771244287491 /loss/FBt2/detection_loss‚0.6030381516265195 2loss/FBt3/assignment_loss‚0.049064720301733665 /loss/FRt1/detection_loss‚2.6684620762243867 /loss/SRqqt1/detection_loss‚2.74575221997041 1loss/SRqqt4/detection_loss‚1.8378805523288662 /loss/FRt1/assignment_loss‚4.506610654294491 0loss/FRt2/assignment_loss‚1.6694335164847198 1loss/SRqqt2/detection_loss‚1.0999198533594607 /loss/FBt4/detection_loss‚2.7395259618759153 $loss/total_loss‚48.2613696165281 _step‚340 0loss/FRt4/assignment_loss‚3.5062023793778767 2loss/SRqqt1/assignment_loss‚2.1342526404903484Ô2Æÿ°š†‚(: §ÃäÔè¨x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.46' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.491999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0= 8r‡‚(:ç §ÃäÔŒõz$ disk.nvme0c0n1.out‚ 17498.8984375 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125' proc.memory.availableMB‚ 172.9921875 proc.cpu.threads‚17 disk.dm-1.in‚7.3125 network.sent‚ 9.308428e+06 disk.sda2.out‚ 0.0390625 disk.sda.out‚ 241.13671875 disk.rbd4.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.90283203125# proc.memory.rssMB‚ 2850.69140625 cpu‚25.27114811050788) disk./.usagePercent‚73.71310695904666 disk.drbd1777.in‚ 1.03515625 disk.dm-0.in‚ 4.9765625 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd3.out‚0 disk.sda3.out‚ 241.09765625$ memory_percent‚98.94413948059082* proc.memory.percent‚17.399239540100098 network.recv‚ 1.44597e+06$ disk./.usageGB‚1098.378562927246" disk.nvme0n1.out‚ 17498.8984375 disk.dm-0.out‚ 17498.8984375 disk.dm-1.out‚ 241.09765625 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 4.9765625 disk.sda2.in‚0! disk.rbd2.out‚146.93701171875¶J½‹ˆ‚(:… ¶ÃäÔ£ò¯x gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.792' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}vIov‰‚(:ë ¶ÃäÔéãàz# proc.memory.rssMB‚ 2850.69140625 disk.rbd0.in‚0 network.recv‚ 1.452075e+06 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9765625 disk.rbd4.in‚0% disk.nvme0c0n1.out‚17598.26953125& proc.memory.availableMB‚ 172.984375* proc.memory.percent‚17.399239540100098 disk.rbd3.out‚0 disk.sda3.out‚ 241.76953125 disk.dm-0.in‚ 4.9765625 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.9765625 cpu‚25.270265351495084# disk.nvme0n1.out‚17598.26953125 disk.drbd1777.in‚ 1.03515625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd1.out‚ 1.90283203125 disk.dm-0.out‚17598.26953125 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda.out‚ 241.80859375 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd3.in‚0 disk.sda3.in‚7.3125! disk.rbd2.out‚146.95263671875 disk.rbd4.out‚0% disk./.usageGB‚1098.3784980773926$ memory_percent‚98.94418716430664 network.sent‚ 9.41655e+06) disk./.usagePercent‚73.71310260691858 disk.sda2.in‚0 disk.dm-1.out‚ 241.76953125›2½âšŠ‚(: ÅÃäÔß°Àx gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚97.627' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.525399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^“` [‹‚(:Ð ÅÃäÔæˆÃ{ network.sent‚ 9.422852e+06) disk./.usagePercent‚73.71310260691858% disk./.usageGB‚1098.3784980773926 disk.rbd3.in‚0 disk.rbd1.out‚ 1.90283203125 disk.sda2.in‚0 disk.rbd4.in‚0% disk.nvme0c0n1.out‚17697.27734375 disk.sda3.out‚242 disk.dm-1.out‚242! disk.rbd2.out‚148.52685546875 proc.memory.availableMB‚173# disk.nvme0n1.out‚17697.27734375 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 242.0390625 disk.sda3.in‚7.3125 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.rbd4.out‚0" memory_percent‚98.944091796875* proc.memory.percent‚17.399239540100098 cpu‚25.269438764302606 proc.cpu.threads‚17 disk.dm-0.out‚17697.27734375 disk.drbd1777.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2850.69140625 network.recv‚ 1.457058e+06 disk.nvme0n1.in‚ 4.9765625 disk.dm-0.in‚ 4.9765625 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 4.9765625ä‹2Œ- _wandbruntime‚4029  _runtime‚4029-ƒàÅ¿ & lr-AdamW/pg1‚0.0006618917932341075 & lr-AdamW/pg2‚0.0006618917932341075 trainer/global_step‚4199 # _timestamp‚1.78842055826146e+09 _runtime‚4029.011406298 _step‚341Ù±™…ÊŽÄ & lr-AdamW/pg1‚0.0006618917932341075 & lr-AdamW/pg2‚0.0006618917932341075 trainer/global_step‚4199 # _timestamp‚1.78842055826146e+09 _runtime‚4029.011406298 _step‚341Õd˜Á2- _wandbruntime‚4029  _runtime‚4029ûîžÀÙO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7770322160402 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157161833843712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7792938160640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73332360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7770098511254 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7931519702528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4199 _step‚342 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109179270 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165977021386752 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73333017 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186040170228224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45196466 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165977039389696 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173769977550336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109179270 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165977021386752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77254857 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚178108674465792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚173769977550336 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173769731343872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186040465086976 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28135894 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186408457 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7792709957120 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28135894 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186408457 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7792709957120 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚173769977550336 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173769731343872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186040465086976 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561Öà«Ê 2’- _wandbruntime‚4030  _runtime‚4030™í[ìé&“ãM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173809906982400 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166015359330816 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73349564 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166015377333760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186083212839424 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28143386 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7771930910854 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178149844162048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186477425 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164970307901138 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚4199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73350209 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _step‚343 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186451727 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173810153202176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77272578 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77272578 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173810153202176 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45206823 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178149820221952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186477425 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7772154560210 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0×>[ô¿&”èM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186451727 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173810153202176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77272578 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77272578 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173810153202176 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45206823 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178149820221952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186477425 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7772154560210 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173809906982400 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166015359330816 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73349564 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166015377333760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186083212839424 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28143386 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7771930910854 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178149844162048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186477425 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164970307901138 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚4199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73350209 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/segme #ñŠ/nt.small_pool.current‚262 _step‚343×j :2•- _wandbruntime‚4030  _runtime‚4030±\¹W–Ñ 0loss/FRt2/assignment_loss‚2.0712363357129306 2loss/SRqqt4/assignment_loss‚2.0912854939699175 1loss/FBt2/assignment_loss‚0.18661391558231236 0loss/SRqqt1/detection_loss‚2.238314905928241 _step‚344 2loss/SRqqt2/assignment_loss‚1.2493978471509994 0loss/FBt3/assignment_loss‚0.2707196464461674 /loss/FRt1/detection_loss‚2.4171789980596965 1loss/SRqqt2/detection_loss‚1.1471137470669217 % _timestamp‚1.7884205593487864e+09 1loss/SRqqt1/assignment_loss‚2.078791709202859 0loss/FBt4/assignment_loss‚0.2859785909143587 .loss/FBt2/detection_loss‚0.552512980795101 _runtime‚4030.098621563 /loss/FRt4/assignment_loss‚4.315802553837949 /loss/FRt2/detection_loss‚1.9791746962329615 /loss/FBt1/assignment_loss‚0.577414845302701 /loss/FRt3/detection_loss‚2.4145330850114215 %loss/total_loss‚48.11575364909636 epoch‚3 2loss/SRqqt3/assignment_loss‚1.8857290862271419 1loss/SRqqt3/detection_loss‚1.8043034560978413 /loss/FBt1/detection_loss‚1.8037188132603963 .loss/FBt3/detection_loss‚3.975788176059723 0loss/FRt1/assignment_loss‚3.8025291813744437 /loss/FRt3/assignment_loss‚3.780382612918286 /loss/FRt4/detection_loss‚2.5641168769110334 trainer/global_step‚4199 0loss/SRqqt4/detection_loss‚2.136461429297924 .loss/FBt4/detection_loss‚2.486654665735033Óôm\—Ö 0loss/FRt2/assignment_loss‚2.0712363357129306 2loss/SRqqt4/assignment_loss‚2.0912854939699175 1loss/FBt2/assignment_loss‚0.18661391558231236 0loss/SRqqt1/detection_loss‚2.238314905928241 _step‚344 2loss/SRqqt2/assignment_loss‚1.2493978471509994 0loss/FBt3/assignment_loss‚0.2707196464461674 /loss/FRt1/detection_loss‚2.4171789980596965 1loss/SRqqt2/detection_loss‚1.1471137470669217 % _timestamp‚1.7884205593487864e+09 1loss/SRqqt1/assignment_loss‚2.078791709202859 0loss/FBt4/assignment_loss‚0.2859785909143587 .loss/FBt2/detection_loss‚0.552512980795101 _runtime‚4030.098621563 /loss/FRt4/assignment_loss‚4.315802553837949 /loss/FRt2/detection_loss‚1.9791746962329615 /loss/FBt1/assignment_loss‚0.577414845302701 /loss/FRt3/detection_loss‚2.4145330850114215 %loss/total_loss‚48.11575364909636 epoch‚3 2loss/SRqqt3/assignment_loss‚1.8857290862271419 1loss/SRqqt3/detection_loss‚1.8043034560978413 /loss/FBt1/detection_loss‚1.8037188132603963 .loss/FBt3/detection_loss‚3.975788176059723 0loss/FRt1/assignment_loss‚3.8025291813744437 /loss/FRt3/assignment_loss‚3.780382612918286 /loss/FRt4/detection_loss‚2.5641168769110334 trainer/global_step‚4199 0loss/SRqqt4/detection_loss‚2.136461429297924 .loss/FBt4/detection_loss‚2.486654665735033Ø}ûlô˜‚(:… ÔÃäÔ¾”šx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?„^÷q™‚(:æ ÔÃäÔñ…Ñz disk.dm-1.in‚7.3125 disk.dm-1.out‚ 242.5703125 disk.drbd1777.out‚0 cpu‚25.268639931821006 network.recv‚ 1.460502e+06 disk.nvme0n1.in‚ 4.9765625 disk.sda3.out‚ 242.5703125 proc.cpu.threads‚17 network.sent‚ 9.428174e+06 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0 disk.sda.in‚ 7.36328125( proc.memory.availableMB‚ 173.83984375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0$ memory_percent‚98.93896579742432 disk.rbd3.in‚0 disk.dm-0.in‚ 4.9765625! disk.rbd2.out‚148.54248046875$ disk.nvme0c0n1.out‚ 17796.7265625 disk.sda3.in‚7.3125# proc.memory.rssMB‚ 2850.69140625* proc.memory.percent‚17.399239540100098 disk.rbd4.out‚0 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0( disk./.usagePercent‚73.7131013268809% disk./.usageGB‚1098.3784790039062" disk.nvme0n1.out‚ 17796.7265625 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚ 242.609375 disk.rbd1.out‚ 1.90283203125 disk.dm-0.out‚ 17796.7265625ù}*ešš‚(: ãÃäÔ›˜®x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.993' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.798599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h˜£l›‚(:á ãÃäÔη–{ network.recv‚ 1.467004e+06 disk.nvme0n1.in‚ 4.9765625 disk.dm-1.in‚7.3125( proc.memory.availableMB‚ 173.80859375# proc.memory.rssMB‚ 2850.69140625( disk./.usagePercent‚73.7131013268809 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 242.9609375 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.dm-0.in‚ 4.9765625* proc.memory.percent‚17.399239540100098 network.sent‚ 9.536687e+06 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 242.9609375# disk.nvme0n1.out‚17796.94921875 disk.dm-0.out‚17796.94921875 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.9765625% disk.nvme0c0n1.out‚17796.94921875 disk.sda1.in‚ 0.05078125 disk.sda.out‚243 disk.rbd1.out‚ 1.90283203125! disk.rbd2.out‚148.66748046875 disk.rbd0.out‚0# memory_percent‚98.9391565322876% disk./.usageGB‚1098.3784790039062 disk.rbd4.in‚0 disk.rbd1.in‚0 cpu‚25.267767182090694 proc.cpu.threads‚17J Nœ‚(:… òÃäÔÚò¯x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.011' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±ä€‚(:õ òÃäÔ’ªîz disk.dm-1.out‚ 243.64453125 disk.rbd1.out‚ 1.90283203125! disk.rbd2.out‚148.68310546875 disk.rbd0.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 173.8046875 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 243.68359375 disk.sda3.in‚7.3125 disk.sda3.out‚ 243.64453125 network.recv‚ 1.470899e+06 disk.rbd1.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2850.69140625) disk./.usagePercent‚73.71311310322749 disk.sda2.out‚ 0.0390625 disk.nvme0c0n1.in‚ 4.9765625 cpu‚25.26703383973547 disk.dm-0.in‚ 4.9765625# disk.dm-0.out‚17896.27294921875 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625$ memory_percent‚98.93918037414551 proc.cpu.threads‚17% disk./.usageGB‚1098.3786544799805& disk.nvme0n1.out‚17896.27294921875 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚17896.27294921875 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125* proc.memory.percent‚17.399239540100098 network.sent‚ 9.542575e+06 disk.nvme0n1.in‚ 4.9765625 disk.rbd4.out‚0ŽžËî2ž- _wandbruntime‚4074  _runtime‚4074mU–ðÇŸÁ & lr-AdamW/pg1‚0.0006574635923591322 & lr-AdamW/pg2‚0.0006574635923591322 trainer/global_step‚4249 % _timestamp‚1.7884206033261442e+09 _runtime‚4074.076064162 _step‚345d1†Ì Æ _runtime‚4074.076064162 _step‚345 & lr-AdamW/pg1‚0.0006574635923591322 & lr-AdamW/pg2‚0.0006574635923591322 trainer/global_step‚4249 % _timestamp‚1.7884206033261442e+09Ù­P÷'2¡- _wandbruntime‚4074  _runtime‚4074J &ß'¢ÙO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚167893936592896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78140809 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188600981 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110460172 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166838688818974 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180167159278592 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚167893918589952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166838929507482 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158976973775872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110434502 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7884837297664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7884837297664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78140809 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78140807 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28460653 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚175778527684096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚175778527684096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚167893936592896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8025023967744 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188575309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7861732082462 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7861955731610 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110460172 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78140807 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7884609094144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7884609094144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8025299080704 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74174869 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110434502 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561Ÿ  ä'£ÞO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚167893918589952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166838929507482 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158976973775872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110434502 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7884837297664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7884837297664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78140809 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78140807 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28460653 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚175778527684096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚175778527684096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚167893936592896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8025023967744 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188575309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7861732082462 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7861955731610 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110460172 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78140807 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7884609094144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7884609094144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8025299080704 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74174869 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110434502 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚167893936592896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78140809 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188600981 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110460172 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166838688818974 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180167159278592 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299Ú1É\ø2¤- _wandbruntime‚4074  _runtime‚4074õâ;zk¥ãM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 % _timestamp‚1.7884206041895914e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188644277 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110485749 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚175818949542400 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚159013276233728 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7863788131418 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚175818949542400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7886446788608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4249 _step‚347 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110485749 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45724571 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStaå×~tsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7863564482062 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45724573 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚4074.940219152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188644277 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78158528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7886675005440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110460053 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167932274536960 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚159013259194368 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78158526 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8027165874176 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28467752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180208328974848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166876823676430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8026888677376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74191646 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚284670759²î&¦èM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188644277 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110485749 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚175818949542400 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚159013276233728 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7863788131418 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚175818949542400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7886446788608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4249 _step‚347 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110485749 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45724571 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7863564482062 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45724573 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚4074.940219152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188644277 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78158528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7886675005440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110460053 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167932274536960 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚159013259194368 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78158526 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8027165874176 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28467752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180208328974848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166876823676430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8026888677376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74191646 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28467075 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 % _timestamp‚1.7884206041895914e+09ÛE¾Å²2§- _wandbruntime‚4074  _runtime‚4074˜²•†\¨Ö 2loss/SRqqt3/assignment_loss‚1.8098287249720373 1loss/SRqqt1/detection_loss‚2.1837239653775185 /loss/FRt2/detection_loss‚1.8682955933565444 0loss/FBt2/detection_loss‚0.43127496246499625 0loss/FRt2/assignment_loss‚1.8135965276848187 0loss/FRt4/assignment_loss‚3.5762135502364902 2loss/SRqqt1/assignment_loss‚2.1971541480584578 1loss/FBt2/assignment_loss‚0.19989834921895555 1loss/FBt3/assignment_loss‚0.25221333517692984 /loss/FRt1/detection_loss‚2.5995116037401287 trainer/global_step‚4249 % _timestamp‚1.7884206041917937e+09 .loss/FRt1/assignment_loss‚4.23966189586755 2loss/SRqqt4/assignment_loss‚1.7080906444058126 1loss/SRqqt3/detection_loss‚2.1135423973549243 /loss/FBt3/detection_loss‚2.7045403957366942 .loss/FBt4/detection_loss‚2.484845995903015 %loss/total_loss‚46.70838383015132 _runtime‚4074.941599242 0loss/FBt4/assignment_loss‚0.5999125166692667 .loss/FBt1/detection_loss‚2.236941589249505 /loss/FRt3/assignment_loss‚3.530887560966687 0loss/FBt1/assignment_loss‚0.6944886164532768 /loss/FRt3/detection_loss‚2.5122657082019706 .loss/FRt4/detection_loss‚2.563837591972616 1loss/SRqqt2/detection_loss‚0.9469298924988424 2loss/SRqqt2/assignment_loss‚1.4151222715810152 1loss/SRqqt4/detection_loss‚2.0256059930032615 epoch‚3 _step‚348š.da©Û _runtime‚4074.941599242 0loss/FBt4/assignment_loss‚0.5999125166692667 .loss/FBt1/detection_loss‚2.236941589249505 /loss/FRt3/assignment_loss‚3.530887560966687 0loss/FBt1/assignment_loss‚0.6944886164532768 /loss/FRt3/detection_loss‚2.5122657082019706 .loss/FRt4/detection_loss‚2.563837591972616 1loss/SRqqt2/detection_loss‚0.9469298924988424 2loss/SRqqt2/assignment_loss‚1.4151222715810152 1loss/SRqqt4/detection_loss‚2.0256059930032615 epoch‚3 _step‚348 2loss/SRqqt3/assignment_loss‚1.8098287249720373 1loss/SRqqt1/detection_loss‚2.1837239653775185 /loss/FRt2/detection_loss‚1.8682955933565444 0loss/FBt2/detection_loss‚0.43127496246499625 0loss/FRt2/assignment_loss‚1.8135965276848187 0loss/FRt4/assignment_loss‚3.5762135502364902 2loss/SRqqt1/assignment_loss‚2.1971541480584578 1loss/FBt2/assignment_loss‚0.19989834921895555 1loss/FBt3/assignment_loss‚0.25221333517692984 /loss/FRt1/detection_loss‚2.5995116037401287 trainer/global_step‚4249 % _timestamp‚1.7884206041917937e+09 .loss/FRt1/assignment_loss‚4.23966189586755 2loss/SRqqt4/assignment_loss‚1.7080906444058126 1loss/SRqqt3/detection_loss‚2.1135423973549243 /loss/FBt3/detection_loss‚2.7045403957366942 .loss/FBt4/detection_loss‚2.484845995903015 %loss/total_loss‚46.70838383015132Ü“$ö²Žª‚(:ƒ ÄäÔ‡ƒ›x gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.26' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.652 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bt«‚(:é ÄäÔÀ™Ëz disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125' proc.memory.availableMB‚ 173.8359375 cpu‚25.266183715311115# disk.dm-0.out‚17995.43115234375 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 network.sent‚ 9.548801e+06 disk.rbd1.out‚ 1.90283203125 disk.sda2.in‚0 disk.rbd2.out‚ 148.77734375 disk.nvme0c0n1.in‚ 4.9765625$ disk./.usageGB‚1098.378833770752( disk.nvme0c0n1.out‚17995.43115234375 disk.sda1.out‚0 disk.sda3.in‚7.3125 proc.cpu.threads‚17) disk./.usagePercent‚73.71312513558159& disk.nvme0n1.out‚17995.43115234375 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.dm-1.out‚244.0625 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚ 244.1015625* proc.memory.percent‚17.399239540100098 network.recv‚ 1.475175e+06 disk.dm-0.in‚ 4.9765625 disk.rbd0.in‚0$ memory_percent‚98.93898963928223# proc.memory.rssMB‚ 2850.69140625 disk.nvme0n1.in‚ 4.9765625 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda3.out‚244.0625ç­4™¬‚(:Ž ÄäÔŒ„x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.72' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.343999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìö¢t­‚(:é ÄäÔ©›¢{ disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0# memory_percent‚98.9384651184082# proc.memory.rssMB‚ 2850.69140625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.out‚ 244.59375 disk.sda2.out‚ 0.0390625 network.recv‚ 1.480304e+06) disk./.usagePercent‚73.71313332782269 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚7.3125 disk.sda.in‚ 7.36328125 network.sent‚ 9.655706e+06& disk.nvme0n1.out‚18095.78466796875# disk.dm-0.out‚18095.78466796875 disk.dm-1.out‚ 244.59375 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0& proc.memory.availableMB‚ 173.921875 disk.sda3.in‚7.3125 disk.rbd1.out‚ 1.90283203125 disk.nvme0n1.in‚ 4.9765625 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9765625* proc.memory.percent‚17.399239540100098 disk.rbd2.out‚ 148.79296875( disk.nvme0c0n1.out‚18095.78466796875 disk.sda.out‚ 244.6328125 cpu‚25.26532777792862% disk./.usageGB‚1098.3789558410645 disk.dm-0.in‚ 4.9765625 disk.sda2.in‚0áˆpB®‚(:… ŸÄäÔÒÒ£x gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÖ½«w¯‚(:ì ŸÄäÔð½é{ disk.sda3.in‚7.3125 disk.sda3.out‚ 245.078125 proc.cpu.threads‚17) disk./.usagePercent‚73.71312129546858 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd0.out‚0 network.recv‚ 1.48466e+06# proc.memory.rssMB‚ 2850.69140625 cpu‚25.264539811064225 disk.nvme0c0n1.in‚ 4.9765625( disk.nvme0c0n1.out‚18096.02294921875 disk.sda.out‚ 245.1171875 disk.rbd2.in‚0$ disk./.usageGB‚1098.378776550293 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 245.078125 disk.rbd2.out‚ 150.3671875 disk.sda1.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚18096.02294921875 disk.sda2.out‚ 0.0390625( proc.memory.availableMB‚ 173.90234375* proc.memory.percent‚17.399239540100098 network.sent‚ 9.661698e+06 disk.drbd1777.in‚ 1.03515625 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.rbd1.out‚ 1.90283203125# disk.dm-0.out‚18096.02294921875$ memory_percent‚98.93858432769775 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 4.9765625 disk.dm-0.in‚ 4.9765625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0du}N2°- _wandbruntime‚4119  _runtime‚4119«­ÝƱÀ trainer/global_step‚4299 $ _timestamp‚1.788420648557252e+09 _runtime‚4119.307191119 _step‚349 & lr-AdamW/pg1‚0.0006529819780611915 & lr-AdamW/pg2‚0.0006529819780611915êJã—˲Š& lr-AdamW/pg1‚0.0006529819780611915 & lr-AdamW/pg2‚0.0006529819780611915 trainer/global_step‚4299 $ _timestamp‚1.788420648557252e+09 _runtime‚4119.307191119 _step‚349Ýä2³- _wandbruntime‚4119  _runtime‚4119çCÆÆ*#´ÙO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75017073 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46231968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168745445282982 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190742161 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _step‚350 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79026757 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111715404 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28785105 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79026759 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28784454 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79026759 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚177787324024320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75016420 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚4299 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160792079629312 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚177787324024320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚169810815793152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚653 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚169810833796096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190767833 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190344137581568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182225644091392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7976736434688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚177787570230784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_sÿ̵plit_bytes.all.current‚ 290664448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79026757 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160792096668672 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚177787570230784 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0º4ä'µÞO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚4299 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160792079629312 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚177787324024320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚169810815793152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚653 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚169810833796096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190767833 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190344137581568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182225644091392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7976736434688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚177787570230784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79026757 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160792096668672 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚177787570230784 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75017073 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46231968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168745445282982 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190742161 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _step‚350 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79026757 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111715404 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28785105 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79026759 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28784454 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79026759 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚177787324024320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75016420 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624Þü 2¶- _wandbruntime‚4120  _runtime‚4120—hƒÃé&·ãM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8120379990528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168783580140438 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160828399126528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177827499662848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7978345925632 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190785431 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160828382087168 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79044476 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46242323 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190811129 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚169849153737216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7955198053270 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190811129 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75033621 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚169849153737216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7978345925632 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 % _timestamp‚1.7884206494282188e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚169849171740160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚190387470974976 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168783820829154 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190785431 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111766651 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8120657187328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7955421702626 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79044478 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177827745882624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79044476 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 _step‚351 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177827745882624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7978574142464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79044478«a‡{î&¸èM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7955421702626 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79044478 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177827745882624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79044476 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 _step‚351 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177827745882624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7978574142464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79044478 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8120379990528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168783580140438 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160828399126528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177827499662848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7978345925632 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190785431 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160828382087168 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79044476 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46242323 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190811129 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚169849153737216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7955198053270 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190811129 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75033621 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚169849153737216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7978345925632 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 % _timestamp‚1.7884206494282188e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚169849171740160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚190387470974976 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168783820829154 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190785431 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111766651 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8120657187328ß³‘±2¹- _wandbruntime‚4120  _runtime‚4120ÝA•ûºÓ 2loss/SRqqt2/assignment_loss‚1.4939276113975466 0loss/FBt1/assignment_loss‚0.2883808184764348 3loss/FBt3/assignment_loss‚0.0007802413383615203 0loss/FRt3/assignment_loss‚3.5926999118593006 0loss/FRt4/assignment_loss‚3.5833006920637906 2loss/SRqqt4/assignment_loss‚1.9518656083222092 /loss/FBt2/detection_loss‚0.7777281875009923 .loss/FBt3/detection_loss‚3.164001762866974 _step‚352 0loss/FBt4/assignment_loss‚0.2891436549834907 .loss/FRt4/detection_loss‚2.252579809890853 1loss/SRqqt2/detection_loss‚0.9988813900241726 1loss/SRqqt3/detection_loss‚2.0930954501396273 %loss/total_loss‚48.91035292892509 trainer/global_step‚4299 .loss/FRt1/assignment_loss‚4.07633119682933 _runtime‚4120.179704672 2loss/SRqqt1/assignment_loss‚2.0611515623681687 1loss/FBt2/assignment_loss‚0.20865200807385947 0loss/SRqqt1/detection_loss‚2.178306896020384 $ _timestamp‚1.788420649429904e+09 .loss/FRt2/detection_loss‚1.723610556899355 /loss/FBt1/detection_loss‚2.9843651711940766 epoch‚3 0loss/FRt2/assignment_loss‚2.2876800894737244 2loss/SRqqt3/assignment_loss‚1.7791771051905505 /loss/FRt3/detection_loss‚2.6803025868203907 -loss/FBt4/detection_loss‚4.07330048084259 /l‘Ê“ª^oss/FRt1/detection_loss‚2.5130100816834804 0loss/SRqqt4/detection_loss‚1.858080054665434ka;^»Ø 2loss/SRqqt1/assignment_loss‚2.0611515623681687 1loss/FBt2/assignment_loss‚0.20865200807385947 0loss/SRqqt1/detection_loss‚2.178306896020384 $ _timestamp‚1.788420649429904e+09 .loss/FRt2/detection_loss‚1.723610556899355 /loss/FBt1/detection_loss‚2.9843651711940766 epoch‚3 0loss/FRt2/assignment_loss‚2.2876800894737244 2loss/SRqqt3/assignment_loss‚1.7791771051905505 /loss/FRt3/detection_loss‚2.6803025868203907 -loss/FBt4/detection_loss‚4.07330048084259 /loss/FRt1/detection_loss‚2.5130100816834804 0loss/SRqqt4/detection_loss‚1.858080054665434 2loss/SRqqt2/assignment_loss‚1.4939276113975466 0loss/FBt1/assignment_loss‚0.2883808184764348 3loss/FBt3/assignment_loss‚0.0007802413383615203 0loss/FRt3/assignment_loss‚3.5926999118593006 0loss/FRt4/assignment_loss‚3.5833006920637906 2loss/SRqqt4/assignment_loss‚1.9518656083222092 /loss/FBt2/detection_loss‚0.7777281875009923 .loss/FBt3/detection_loss‚3.164001762866974 _step‚352 0loss/FBt4/assignment_loss‚0.2891436549834907 .loss/FRt4/detection_loss‚2.252579809890853 1loss/SRqqt2/detection_loss‚0.9988813900241726 1loss/SRqqt3/detection_loss‚2.0930954501396273 %loss/total_loss‚48.91035292892509 trainer/global_step‚4299 .loss/FRt1/assignment_loss‚4.07633119682933 _runtime‚4120.179704672ày?Ñò¼‚(:… ®ÄäÔæ¾—x gpu.0.gpu‚18.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.011' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0οz½‚(:ï ®ÄäÔ£ˆ{% disk./.usageGB‚1098.3788375854492 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125) disk./.usagePercent‚73.71312539158913 disk.rbd3.out‚0" disk.dm-0.out‚18195.4833984375 network.sent‚ 9.667695e+06 network.recv‚ 1.488832e+06 disk.dm-1.in‚7.3125 disk.sda3.out‚ 245.62890625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9765625* proc.memory.percent‚17.399239540100098 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd1.out‚ 1.90283203125 disk.dm-0.in‚ 4.9765625$ memory_percent‚98.93975257873535' proc.memory.availableMB‚ 173.7109375 disk.drbd1777.in‚ 1.03515625 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 4.9765625# proc.memory.rssMB‚ 2850.69140625 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 cpu‚25.263774113189314 disk.dm-1.out‚ 245.62890625 disk.rbd2.out‚ 150.3828125 disk.rbd4.in‚0 disk.sda.out‚ 245.66796875% disk.nvme0n1.out‚18195.4833984375 proc.cpu.threads‚17 disk.rbd0.in‚0' disk.nvme0c0n1.out‚18195.4833984375„:™›¾‚(: ½ÄäÔí‚“x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.171' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.834200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­Ö‘v¿‚(:ë ½ÄäÔÑò¸z disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 246.09375" disk.dm-0.out‚18294.5341796875 disk.dm-1.in‚7.3125! disk.rbd2.out‚150.47705078125 disk.nvme0c0n1.in‚ 4.9765625 disk.sda.out‚ 246.1328125 disk.rbd1.out‚ 2.05126953125 cpu‚25.26295235216933) disk./.usagePercent‚73.71312129546858 disk.rbd0.in‚0 disk.rbd4.in‚0* proc.memory.percent‚17.399239540100098 network.recv‚ 1.495884e+06 disk.dm-1.out‚ 246.09375 disk.rbd1.in‚0 network.sent‚ 9.776413e+06 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.nvme0n1.in‚ 4.9765625$ memory_percent‚98.93982410430908# proc.memory.rssMB‚ 2850.69140625$ disk./.usageGB‚1098.378776550293 disk.dm-0.in‚ 4.9765625 disk.sda1.out‚0( proc.memory.availableMB‚ 173.69921875% disk.nvme0n1.out‚18294.5341796875 disk.sda2.in‚0 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚18294.5341796875 disk.rbd3.out‚0 proc.cpu.threads‚17úyâ•™À‚(:Ž ÌÄäÔ¸„›x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚95.51' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.102000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÌ0Ì€Á‚(:õ ÌÄäÔ…‘·z network.recv‚ 1.49928e+06 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 246.69921875$ memory_percent‚98.94130229949951 network.sent‚ 9.781695e+06% disk./.usageGB‚1098.3787803649902& disk.nvme0n1.out‚18393.93212890625 disk.drbd1777.in‚ 1.03515625 disk.rbd1.in‚0 disk.rbd1.out‚ 2.05126953125( disk.nvme0c0n1.out‚18393.93212890625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.in‚7.3125( proc.memory.availableMB‚ 173.45703125* proc.memory.percent‚17.399239540100098# disk.dm-0.out‚18393.93212890625 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 246.69921875 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9765625# proc.memory.rssMB‚ 2850.69140625 disk.nvme0n1.in‚ 4.9765625 disk.drbd1777.out‚0 disk.dm-0.in‚ 4.9765625 disk.dm-1.in‚7.3125 disk.rbd3.out‚0 disk.sda.out‚ 246.73828125! disk.rbd2.out‚150.48876953125 disk.rbd0.in‚0 disk.sda.in‚ 7.36328125 cpu‚25.262188587622745 proc.cpu.threads‚17( disk./.usagePercent‚73.7131215514761 disk.rbd0.out‚0 disk.rbd4.in‚0}^'2Â- _wandbruntime‚4164  _runtime‚4164圿ÇÃÁ & lr-AdamW/pg1‚0.0006484478843019877 & lr-AdamW/pg2‚0.0006484478843019877 trainer/global_step‚4349 % _timestamp‚1.7884206938965077e+09 _runtime‚4164.646427333 _step‚353æy bÌÄÆ & lr-AdamW/pg1‚0.0006484478843019877 & lr-AdamW/pg2‚0.0006484478843019877 trainer/global_step‚4349 % _timestamp‚1.7884206938965077e+09 _runtime‚4164.646427333 _step‚353ᆟM2Å- _wandbruntime‚4164  _runtime‚4164 ½âŠß'ÆÙO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79912709 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29111623 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚192496144894464 % _timestamp‚1.7884206938972306e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75860685 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8044999224878 _runtime‚4164.647414813 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79912707 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 192934685 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚171727712996352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113021976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46749718 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179796120364544 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112996306 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179796120364544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112996306 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8212321334784 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179796366571008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8068635571712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184284104964096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281404416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8068407368192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚170652201746990 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29110969 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75861341 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8068407368192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184284128904192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192496450238976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79912709 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113021976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚171727730999296 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179796366571008 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚171727712996352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8068635571712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚354 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944À· ä'ÇÞO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75860685 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8044999224878 _runtime‚4164.647414813 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79912707 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 192934685 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚171727712996352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113021976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46749718 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179796120364544 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112996306 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179796120364544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112996306 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8212321334784 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179796366571008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8068635571712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184284104964096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281404416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8068407368192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚170652201746990 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29110969 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75861341 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8068407368192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184284128904192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192496450238976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79912709 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113021976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚171727730999296 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179796366571008 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚171727712996352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8068635571712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚354 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79912709 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29111623 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚192496144894464 % _timestamp‚1.7884206938972306e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992âÔ`an2È- _wandbruntime‚4165  _runtime‚4165;ü†´ÉãM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75877282 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79930426 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8070245062656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192952283 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8214184289280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192952283 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113021857 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29117836 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192977981 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79930428 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚171766068943360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚355 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚171766050940416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46760073 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46760071 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 283488256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚171766068943360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8070473279488 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192539175461376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75877909 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚192539482889728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4165.509080915 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192977981 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179836542222848 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113047553 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚162643522019328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚171766050940416 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113021857 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184325298600448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚170690336604446 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8047055273834 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79930426 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561™~ã|î&ÊèM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884206947582428e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 283488256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚171766068943360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8070473279488 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192539175461376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75877909 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚192539482889728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4165.509080915 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192977981 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179836542222848 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113047553 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚162643522019328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚171766050940416 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113021857 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184325298600448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚170690336604446 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8047055273834 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79930426 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75877282 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79930426 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8070245062656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192952283 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8214184289280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192952283 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113021857 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29117836 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192977981 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79930428 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚171766068943360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚355 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚171766050940416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46760073 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46760071 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181804916704768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47267466 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚164422325414912 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8136856445234 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195101537 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114302878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114277208 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173644610199552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173644628202496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚358 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚172559198899506 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76706417 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194648012420608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76705754 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47267468 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29438288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8305422643712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80798659 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80798659 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181804916704768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47267466 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚164422325414912 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8136856445234 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195101537 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114302878 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114277208 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173644610199552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173644628202496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚358 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚172559198899506 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76706417 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194648012420608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663æØÃgu2Ú- _wandbruntime‚4210  _runtime‚4210€T¿æ&ÛàM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76723126 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194691048560128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80816376 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47277821 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚164458627872768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195144833 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173682948143616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚194691343405568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚186383783413248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788420740066135e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172597333757170 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114302759 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186383759473152 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 _step‚359 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181845338563072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8162144199680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8162144199680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚4399 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181845092343296 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114328455 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29445303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173682966146560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8162372416512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195119135 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181845338563072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8307289086976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172597093068454 _runtime‚ 4210.8165638 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76722462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80816378 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29444641 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173682948143616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8307559992320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚164458644912128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8138465195686 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195119135 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47277823 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114328455 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8162372416512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195144833 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80816376 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114302759 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1Þ:éë&ÜåM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195119135 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181845338563072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8307289086976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172597093068454 _runtime‚ 4210.8165638 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76722462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80816378 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29444641 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173682948143616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8307559992320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚164458644912128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8138465195686 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195119135 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47277823 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114328455 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8162372416512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195144833 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80816376 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114302759 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76723126 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194691048560128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80816376 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47277821 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚164458627872768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195144833 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173682948143616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚194691343405568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚186383783413248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788420740066135e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172597333757170 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114302759 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186383759473152 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 _step‚359 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181845338563072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8162144199680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8162144199680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚4399 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181845092343296 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114328455 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29445303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173682966146560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8162372416512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673ç# o2Ý- _wandbruntime‚4210  _runtime‚4210—¨ŒýXÞÒ .loss/FRt1/assignment_loss‚4.24355918584868 1loss/SRqqt1/detection_loss‚2.5124802037521645 _step‚360 0loss/FRt2/assignment_loss‚1.8186830001718857 1loss/FBt3/assignment_loss‚0.28696604288415983 1loss/SRqqt2/detection_loss‚0.8626676025903888 /loss/FBt2/detection_loss‚0.6909068075337392 0loss/FBt4/assignment_loss‚0.1846362319076434 /loss/FRt4/detection_loss‚2.0756674643526876 /loss/FBt4/detection_loss‚2.8099775314331055 epoch‚3 2loss/SRqqt3/assignment_loss‚2.1420259237923522 1loss/FBt1/assignment_loss‚0.49205763895151905 %loss/total_loss‚47.76811748196112 0loss/FRt4/assignment_loss‚3.9843204744318697 1loss/FBt2/assignment_loss‚0.26480459326600586 .loss/FRt3/detection_loss‚2.555382890360696 0loss/SRqqt4/detection_loss‚1.896281253150169 /loss/FBt1/detection_loss‚2.3556320011615752 /loss/FBt3/detection_loss‚3.2546471878886223 _runtime‚4210.817676386 2loss/SRqqt2/assignment_loss‚1.2585193651935294 /loss/FRt1/detection_loss‚2.4986191395410273 /loss/SRqqt3/detection_loss‚2.15515909296401 /loss/FRt3/assignment_loss‚4.034595180125463 2loss/SRqqt4/assignment_loss‚1.6203022621413494 /loss/FRt2/detection_loss‚1.6102791434409571 $ _timestamp‚1.788420740067863e+09 0loss/SRqqt1/assignment_loss‚2.15994726507752 trainer/global_step‚4399ãÿû´Õß× /loss/FBt2/detection_loss‚0.6909068075337392 0loss/FBt4/assignment_loss‚0.1846362319076434 /loss/FRt4/detection_loss‚2.0756674643526876 /loss/FBt4/detection_loss‚2.8099775314331055 epoch‚3 2loss/SRqqt3/assignment_loss‚2.1420259237923522 1loss/FBt1/assignment_loss‚0.49205763895151905 %loss/total_loss‚47.76811748196112 0loss/FRt4/assignment_loss‚3.9843204744318697 1loss/FBt2/assignment_loss‚0.26480459326600586 .loss/FRt3/detection_loss‚2.555382890360696 0loss/SRqqt4/detection_loss‚1.896281253150169 /loss/FBt1/detection_loss‚2.3556320011615752 /loss/FBt3/detection_loss‚3.2546471878886223 _runtime‚4210.817676386 2loss/SRqqt2/assignment_loss‚1.2585193651935294 /loss/FRt1/detection_loss‚2.4986191395410273 /loss/SRqqt3/detection_loss‚2.15515909296401 /loss/FRt3/assignment_loss‚4.034595180125463 2loss/SRqqt4/assignment_loss‚1.6203022621413494 /loss/FRt2/detection_loss‚1.6102791434409571 $ _timestamp‚1.788420740067863e+09 0loss/SRqqt1/assignment_loss‚2.15994726507752 trainer/global_step‚4399 .loss/FRt1/assignment_loss‚4.24355918584868 1loss/SRqqt1/detection_loss‚2.5124802037521645 _step‚360 0loss/FRt2/assignmjɈent_loss‚1.8186830001718857 1loss/FBt3/assignment_loss‚0.28696604288415983 1loss/SRqqt2/detection_loss‚0.8626676025903888è:…¹Là‚(:… ˆÅäÔÚÈx gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.584' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G;6“á‚(:ô ˆÅäÔâÚ€{ disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125* proc.memory.percent‚17.399239540100098 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda3.out‚ 249.23046875 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 2.146484375 disk.nvme0n1.in‚ 4.9765625$ memory_percent‚98.94163608551025 proc.cpu.threads‚17( disk.nvme0c0n1.out‚18593.16943359375 network.sent‚ 9.907771e+06 disk.sda1.out‚0( proc.memory.availableMB‚ 173.40234375 disk.rbd2.in‚0# disk.dm-0.out‚18593.16943359375# proc.memory.rssMB‚ 2850.69140625! disk.rbd2.out‚152.28173828125 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 249.26953125& disk.nvme0n1.out‚18593.16943359375 disk.dm-0.in‚ 4.9765625 disk.sda2.out‚ 0.0390625% disk./.usageGB‚1098.3792686462402 disk.drbd1777.out‚0 disk.dm-1.in‚7.3125 network.recv‚ 1.517826e+06( disk./.usagePercent‚73.7131543204405 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.dm-1.out‚ 249.23046875 cpu‚25.259179799802432 disk.rbd0.out‚0€ýfüâ‚(:… —ÅäÔÈ—Šx gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…–KÀ|ã‚(:ñ —ÅäԈόz disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.399239540100098# disk.dm-0.out‚18692.25537109375 disk.sda2.out‚ 0.0390625 network.sent‚ 1.0016342e+07( disk./.usagePercent‚73.7131543204405( disk.nvme0c0n1.out‚18692.25537109375 disk.rbd0.out‚0$ memory_percent‚98.94192218780518( proc.memory.availableMB‚ 173.35546875 proc.cpu.threads‚17 disk.sda.out‚ 249.71875 disk.rbd1.in‚0 disk.rbd1.out‚ 2.20947265625 disk.sda1.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.dm-1.out‚ 249.6796875# proc.memory.rssMB‚ 2850.69140625 cpu‚25.258395194293424 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.nvme0n1.in‚ 4.9765625 disk.rbd3.out‚0 network.recv‚ 1.524441e+06 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0& disk.nvme0n1.out‚18692.25537109375 disk.dm-0.in‚ 4.9765625% disk./.usageGB‚1098.3792686462402 disk.rbd3.in‚0 disk.sda3.out‚ 249.6796875 disk.sda2.in‚0 disk.rbd2.out‚152.3720703125W€ ¾šä‚(: ¦ÅäÔ¿§™x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.971999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©Ùù•{å‚(:ð ¦ÅäÔŒ•»{& disk.nvme0n1.out‚18791.96728515625 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda.out‚ 250.3671875 network.recv‚ 1.535583e+06) disk./.usagePercent‚73.71317070492267 disk.rbd0.out‚0 network.sent‚ 1.0025707e+07# disk.dm-0.out‚18791.96728515625 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda3.in‚7.3125 disk.sda3.out‚ 250.328125" memory_percent‚98.941969871521 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd2.out‚152.3876953125 disk.nvme0c0n1.in‚ 4.9765625 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 2.20947265625( proc.memory.availableMB‚ 173.34765625# proc.memory.rssMB‚ 2850.69140625 disk.dm-1.out‚ 250.328125 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚18791.96728515625 cpu‚25.257705209427797 proc.cpu.threads‚17 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0% disk./.usageGB‚1098.3795127868652 disk.nvme0n1.in‚ 4.9765625 disk.dm-0.in‚ 4.9765625 disk.rbd0.in‚0* proc.memory.percent‚17.399239540100098 Ÿ·Û2æ- _wandbruntime‚4255  _runtime‚4255åçÈÑÇçÁ & lr-AdamW/pg1‚0.0006392260487328911 & lr-AdamW/pg2‚0.0006392260487328911 trainer/global_step‚4449 % _timestamp‚1.7884207843687074e+09 _runtime‚4255.118629532 _step‚361%fDIÌèÆ & lr-AdamW/pg1‚0.0006392260487328911 & lr-AdamW/pg2‚0.0006392260487328911 trainer/global_step‚4449 % _timestamp‚1.7884207843687074e+09 _runtime‚4255.118629532 _step‚361éE3:`2é- _wandbruntime‚4255  _runtime‚4255w•®Aß'êÙO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197242717 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77551822 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188401098529792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚188401074589696 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115558110 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183813959251456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 _step‚362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8252205642240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197268389 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47785218 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166237465347072 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 trainer/global_step‚4449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166237448307712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81684607 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 % _timestamp‚1.7884207843696368e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8252205642240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174465714675006 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115583780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29765957 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81684609 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174465955363514 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8228266367294 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚175561525405696 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115558110 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚647 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8252433845760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8398921648128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4255.119869217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183813713044992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81684609 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8228490016442 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8399203052544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183813713044992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚175561507402752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196800301582336Œ7:ä'ëÞO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8252205642240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197268389 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47785218 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166237465347072 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 trainer/global_step‚4449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166237448307712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81684607 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 % _timestamp‚1.7884207843696368e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8252205642240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174465714675006 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115583780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29765957 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81684609 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174465955363514 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8228266367294 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚175561525405696 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115558110 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚647 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8252433845760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8398921648128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4255.119869217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183813713044992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81684609 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8228490016442 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8399203052544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183813713044992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚175561507402752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196800301582336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197242717 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77551822 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188401098529792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚188401074589696 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115558110 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183813959251456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 _step‚362êÙª‘¿2ì- _wandbruntime‚4255  _runtime‚4255—hÝ óíãM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚175599863349760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8254271553536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29773105 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8230322416250 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115583661 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 197285987 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81702328 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884207852237494e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 trainer/global_step‚4449 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81702328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183854134903296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29772469 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8401063582208 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚175599863349760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166273767804928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115609357 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.currenvʇ´ö t‚25696 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚175599845346816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197311685 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77568040 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚183854134903296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197285987 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115609357 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚196843331808256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166273750765568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188442244285952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0±Üºî&îèM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 trainer/global_step‚4449 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81702328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183854134903296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29772469 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8401063582208 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚175599863349760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166273767804928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115609357 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚175599845346816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197311685 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77568040 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚183854134903296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197285987 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115609357 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚196843331808256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166273750765568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188442244285952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚175599863349760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8254271553536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29773105 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8230322416250 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115583661 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 197285987 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81702328 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884207852237494e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428ë­Ýõ2ï- _wandbruntime‚4255  _runtime‚4255òuZðÔ 1loss/SRqqt3/assignment_loss‚2.008625224977732 0loss/FBt1/assignment_loss‚0.4539150955776374 _runtime‚4255.975474439 0loss/FRt1/assignment_loss‚4.0523665144636825 2loss/SRqqt2/assignment_loss‚1.3451897852843808 .loss/FRt3/detection_loss‚2.986522095984426 1loss/SRqqt4/detection_loss‚1.9194307270646096 /loss/FBt1/detection_loss‚2.1711883544921875 /loss/FBt4/detection_loss‚2.9554028312365213 /loss/FRt1/detection_loss‚2.5307980665081256 1loss/SRqqt2/detection_loss‚1.0727533009504118 /loss/FBt3/detection_loss‚2.5999337136745453 0loss/FRt4/assignment_loss‚3.3539974838495255 2loss/SRqqt4/assignment_loss‚1.7984407958388329 /loss/FBt3/assignment_loss‚0.092244949482847 0loss/SRqqt1/detection_loss‚2.478892382038267 1loss/SRqqt3/detection_loss‚2.2480175837874414 trainer/global_step‚4449 0loss/FBt2/assignment_loss‚0.1632994566778885 epoch‚3 $ _timestamp‚1.788420785225667e+09 _step‚364 /loss/FRt3/assignment_loss‚4.478848921841588 1loss/SRqqt1/assignment_loss‚1.889729735098387 .loss/FRt4/detection_loss‚2.320791879296409 /loss/FBt2/detection_loss‚0.5034033228634017 %loss/total_loss‚46.66037937384509 0loss/FRt2/assignment_loss‚1.7182300984859467 3loss/FBt4/assignment_loss‚0.0033612757009298853 /loss/FRt2/detection_loss‚1.5149957786693617C‹;t_ñÙ 1loss/SRqqt3/assignment_loss‚2.008625224977732 0loss/FBt1/assignment_loss‚0.4539150955776374 _runtime‚4255.975474439 0loss/FRt1/assignment_loss‚4.0523665144636825 2loss/SRqqt2/assignment_loss‚1.3451897852843808 .loss/FRt3/detection_loss‚2.986522095984426 1loss/SRqqt4/detection_loss‚1.9194307270646096 /loss/FBt1/detection_loss‚2.1711883544921875 /loss/FBt4/detection_loss‚2.9554028312365213 /loss/FRt1/detection_loss‚2.5307980665081256 1loss/SRqqt2/detection_loss‚1.0727533009504118 /loss/FBt3/detection_loss‚2.5999337136745453 0loss/FRt4/assignment_loss‚3.3539974838495255 2loss/SRqqt4/assignment_loss‚1.7984407958388329 /loss/FBt3/assignment_loss‚0.092244949482847 0loss/SRqqt1/detection_loss‚2.478892382038267 1loss/SRqqt3/detection_loss‚2.2480175837874414 trainer/global_step‚4449 0loss/FBt2/assignment_loss‚0.1632994566778885 epoch‚3 $ _timestamp‚1.788420785225667e+09 _step‚364 /loss/FRt3/assignment_loss‚4.478848921841588 1loss/SRqqt1/assignment_loss‚1.889729735098387 .loss/FRt4/detection_loss‚2.320791879296409 /loss/FBt2/detection_loss‚0.5034033228634017 %loss/total_loss‚46.66037937384509 0loss/FRt2/assignment_loss‚1.7182300984859467 3loss/FBt4/assignment_loss‚0.0033612757009298853 /loss/FRt2/detection_loss‚1.5149957786693617ìÞmmŽò‚(:ƒ µÅäÔ‹Û©x gpu.0.gpu‚18.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.37' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø&’áyó‚(:î µÅäÔÒí{ disk.dm-0.in‚ 4.9765625# disk.dm-0.out‚18891.03369140625 disk.rbd1.in‚0* proc.memory.percent‚17.399239540100098% disk./.usageGB‚1098.3795928955078 disk.sda2.out‚ 0.0390625( disk.nvme0c0n1.out‚18891.03369140625 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda3.out‚ 250.8046875 proc.cpu.threads‚17 network.sent‚ 1.0032051e+07 disk.sda2.in‚0 disk.rbd2.out‚152.4501953125 disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.rbd3.out‚0 disk.sda.out‚ 250.84375 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.256973535882494 network.recv‚ 1.540277e+06 disk.dm-1.out‚ 250.8046875 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd1.out‚ 2.20947265625 disk.drbd1777.out‚0 disk.rbd4.in‚0# proc.memory.rssMB‚ 2850.69140625( disk./.usagePercent‚73.7131760810809 disk.dm-1.in‚7.3125 disk.rbd4.out‚0 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚ 4.9765625& disk.nvme0n1.out‚18891.03369140625$ memory_percent‚98.94237518310547% proc.memory.availableMB‚ 173.28125k¥Äoô‚(:… ÄÅäÔÇíºx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶öy†|õ‚(:ñ ÄÅäÔ£Š÷z% disk./.usageGB‚1098.3802490234375 disk.rbd3.out‚0( disk./.usagePercent‚73.7132201143768 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚ 4.9765625( proc.memory.availableMB‚ 173.03515625 disk.rbd2.out‚153.9775390625 disk.drbd1777.in‚ 1.03515625% disk.nvme0n1.out‚18892.8798828125 cpu‚25.2562048460066 disk.dm-0.in‚ 4.9765625 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 2.20947265625 network.sent‚ 1.0139763e+07' disk.nvme0c0n1.out‚18892.8798828125 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 251.37890625* proc.memory.percent‚17.399239540100098 network.recv‚ 1.545671e+06" disk.dm-0.out‚18892.8798828125 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0$ memory_percent‚98.94387722015381# proc.memory.rssMB‚ 2850.69140625 disk.dm-1.out‚ 251.37890625 disk.sda.out‚ 251.41796875 V{H›ö‚(: ÓÅäÔª˜x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾y1y÷‚(:î ÓÅäÔéâªz disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda2.out‚ 0.0390625 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0% disk.nvme0n1.out‚18991.9111328125* proc.memory.percent‚17.399239540100098( disk./.usagePercent‚73.7132201143768 disk.sda3.out‚ 251.8515625" disk.dm-0.out‚18991.9111328125 disk.dm-1.in‚7.3125$ memory_percent‚98.94092082977295 disk.rbd3.in‚0 disk.sda.out‚ 251.890625# proc.memory.rssMB‚ 2850.69140625 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 173.51953125 network.sent‚ 1.0145827e+07 disk.rbd4.out‚0 disk.sda3.in‚7.3125 disk.dm-0.in‚ 4.9765625 disk.dm-1.out‚ 251.8515625 disk.rbd0.out‚0 network.recv‚ 1.549863e+06% disk./.usageGB‚1098.3802490234375 disk.drbd1777.out‚0 disk.rbd4.in‚0' disk.nvme0c0n1.out‚18991.9111328125 disk.nvme0n1.in‚ 4.9765625 disk.rbd2.in‚0 cpu‚25.25543760307905 disk.rbd1.out‚ 2.20947265625 disk.rbd2.out‚154.0400390625 disk.rbd0.in‚0,™ƒÊ2ø- _wandbruntime‚4300  _runtime‚4300–opÇùÁ _runtime‚4300.343587605 _step‚365 & lr-AdamW/pg1‚0.0006345402287397088 & lr-AdamW/pg2‚0.0006345402287397088 trainer/global_step‚4499 % _timestamp‚1.7884208295936737e+09nv !ÌúÆ & lr-AdamW/pg2‚0.0006345402287397088 trainer/global_step‚4499 % _timestamp‚1.7884208295936737e+09 _runtime‚4300.343587605 _step‚365 & lr-AdamW/pg1‚0.0006345402287397088íXî€2û- _wandbruntime‚4300  _runtime‚4300ð±ˆcÞ'üØO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177478422608896 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚168052588239872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185822509385216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48302968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8320123587650 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30095369 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82570557 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177478404605952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8344332982784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚177478422608896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190459559402496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _step‚366 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116864682 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177478404605952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116864682 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185822509385216 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116839012 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78398337 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8492617288704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82570557 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168052571200512 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8344104779264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82570559 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚176372471139014 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8319899938502 $ _timestamp‚1.788420829594391e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198952200631296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8344332982784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8344104779264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0QF¥ ¾ýÝO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82570559 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199435241 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48302966 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190459583342592 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199409569 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185822755591680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185822755591680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198951907869696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 M@DeviceStatsMonitor.on_train_batch_bZ%"start/inactive_split.all.freed‚78397679 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚4300.344499322 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116839012 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82570559 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚176372471139014 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8319899938502 $ _timestamp‚1.788420829594391e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198952200631296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8344332982784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8344104779264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177478422608896 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚168052588239872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185822509385216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48302968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8320123587650 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30095369 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82570557 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177478404605952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8344332982784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚177478422608896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190459559402496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _step‚366 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116864682 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177478404605952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116864682 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185822509385216 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116839012 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78398337 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8492617288704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82570557 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168052571200512 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8344104779264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199452839 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82588276 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185862931243520 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190500753038848 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚176410605996470 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 _runtime‚4301.198976019 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185862931243520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190500729098752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8346170690560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78415573 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8494490567168 % _timestamp‚1.7884208304483473e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199478537 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82588276 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚177516760552960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8346170690560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177516760552960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8345942473728 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚176410846685186 _step‚367 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8345942473728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚177516742550016 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199478537 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚168088890697728 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8321955987458 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198995243606016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78414932 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8494211273216 trainer/global_step‚4499 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82588278 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30101611 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168088873658368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8321732338102 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚177516742550016F§×Úî&€èM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8321732338102 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚177516742550016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48313323 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48313321 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198994940371968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199452839 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82588276 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185862931243520 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190500753038848 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚176410605996470 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 _runtime‚4301.198976019 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185862931243520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190500729098752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8346170690560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78415573 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8494490567168 % _timestamp‚1.7884208304483473e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199478537 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82588276 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚177516760552960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8346170690560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177516760552960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8345942473728 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚176410846685186 _step‚367 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8345942473728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚177516742550016 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199478537 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚168088890697728 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8321955987458 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198995243606016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78414932 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8494211273216 trainer/global_step‚4499 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82588278 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30101611 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168088873658368ïÜ%0¶2- _wandbruntime‚4301  _runtime‚4301Î º”Y‚Ó 0loss/FBt1/assignment_loss‚0.4145257054724425 /loss/FBt1/detection_loss‚3.2508287906646727 0loss/FRt2/assignment_loss‚2.5547737739302896 0loss/FBt3/assignment_loss‚0.5484532217184702 0loss/FBt4/assignment_loss‚0.7220548371551558 /loss/FBt4/detection_loss‚2.4618457183241844 epoch‚3 1loss/SRqqt2/detection_loss‚0.7832787838336583 0loss/SRqqt3/detection_loss‚1.904773466966369 %loss/total_loss‚49.72211186301523 /loss/FRt1/assignment_loss‚4.730566624232701 2loss/SRqqt1/assignment_loss‚1.8318377638856569 2loss/SRqqt3/assignment_loss‚2.1045072325928644 /loss/FRt2/detection_loss‚1.7063934473828837 .loss/FRt3/detection_loss‚2.646747560877549 1loss/SRqqt1/detection_loss‚2.4851053337256115 trainer/global_step‚4499 _step‚368 0loss/FRt3/assignment_loss‚3.6749042523534676 0loss/FRt4/assignment_loss‚3.0888972439263997 2loss/SRqqt2/assignment_loss‚1.5913891818475079 .loss/FRt4/detection_loss‚2.381908910839181 1loss/SRqqt4/detection_loss‚1.8277209899548827 _runtime‚4301.199999307 2loss/SRqqt4/assignment_loss‚1.5156454783061455 .loss/FRt1/detection_loss‚2.659789053457124 /loss/FBt2/detection_loss‚0.7045021938853139 .loss/FBt3/detection_loss‚3.948248545328776 0loss/FBt2/assignment_loss‚0.1834137523539174 % _timestamp‚1.7884208304501822e+09uŸ=^ƒØ % _timestamp‚1.7884208304501822e+09 0loss/FBt1/assignment_loss‚0.4145257054724425 /loss/FBt1/detection_loss‚3.2508287906646727 0loss/FRt2/assignment_loss‚2.5547737739302896 0loss/FBt3/assignment_loss‚0.5484532217184702 0loss/FBt4/assignment_loss‚0.7220548371551558 /loss/FBt4/detection_loss‚2.4618457183241844 epoch‚3 1loss/SRqqt2/detection_loss‚0.7832787838336583 0loss/SRqqt3/detection_loss‚1.904773466966369 %loss/total_loss‚49.72211186301523 /loss/FRt1/assignment_loss‚4.730566624232701 2loss/SRqqt1/assignment_loss‚1.8318377638856569 2loss/SRqqt3/assignment_loss‚2.1045072325928644 /loss/FRt2/detection_loss‚1.7063934473828837 .loss/FRt3/detection_loss‚2.646747560877549 1loss/SRqqt1/detection_loss‚2.4851053337256115 trainer/global_step‚4499 _step‚368 0loss/FRt3/assignment_loss‚3.6749042523534676 0loss/FRt4/assignment_loss‚3.0888972439263997 2loss/SRqqt2/assignment_loss‚1.5913891818475079 .loss/FRt4/detection_loss‚2.381908910839181 1loss/SRqqt4/detection_loss‚1.8277209899548827 _runtime‚4301.199999307 2loss/SRqqt4/assignment_loss‚1.5156454783061455 .loss/FRt1/detection_loss‚2.659789053457124 /loss/FBt2/detection_loss‚0.7045021938853139 .loss/FBt3/detection_loss‚3.948248545328776 0loss/FBt2/assignment_loss‚0.1834137523539174ð$¹™„‚(:Ž âÅäÔ¯¨Žx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.44' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.887999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üŽÇ^…‚(:ï âÅäÔå×´z disk.rbd3.out‚0( disk./.usagePercent‚73.7132201143768 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0# proc.memory.rssMB‚ 2850.69140625* proc.memory.percent‚17.399239540100098 disk.sda2.out‚ 0.0390625# disk.dm-0.out‚19091.27001953125 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda.out‚252.4375( proc.memory.availableMB‚ 173.04296875 cpu‚25.25472524011262 disk.dm-1.out‚ 252.3984375 disk.rbd1.out‚ 2.20947265625 disk.nvme0n1.in‚ 4.9765625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.9765625$ memory_percent‚98.94382953643799 proc.cpu.threads‚17 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.dmÿYñTi-0.in‚ 4.9765625 disk.rbd2.out‚154.0556640625 disk.drbd1777.out‚0& disk.nvme0n1.out‚19091.27001953125( disk.nvme0c0n1.out‚19091.27001953125 disk.sda3.in‚7.3125 disk.sda3.out‚ 252.3984375 disk.dm-1.in‚7.3125 network.sent‚ 1.0151757e+07 network.recv‚ 1.554031e+06% disk./.usageGB‚1098.3802490234375 disk.rbd1.in‚0ÞÙÊš†‚(: ñÅäÔŸ½x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.57' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.913999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R›&z‡‚(:ï ñÅäÔÒ»z disk.rbd4.out‚0 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.dm-1.in‚7.3125# proc.memory.rssMB‚ 2850.69140625 disk.rbd3.in‚0 disk.rbd1.out‚ 2.20947265625 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 253.078125 cpu‚25.254018063553723 network.recv‚ 1.56053e+06( disk./.usagePercent‚73.7132201143768 disk.sda3.in‚7.3125 disk.sda3.out‚ 253.078125 disk.nvme0n1.in‚ 4.9765625 network.sent‚ 1.0260493e+07 disk.drbd1777.in‚ 1.03515625 disk.dm-0.in‚ 4.9765625 disk.sda2.in‚0! disk.rbd2.out‚154.21240234375 proc.cpu.threads‚17( disk.nvme0c0n1.out‚19190.44775390625 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 253.1171875 disk.rbd1.in‚0& disk.nvme0n1.out‚19190.44775390625 disk.rbd0.in‚0# memory_percent‚98.9443302154541 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd2.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.399239540100098 disk.drbd1777.out‚0' proc.memory.availableMB‚ 172.9609375% disk./.usageGB‚1098.3802490234375# disk.dm-0.out‚19190.44775390625 disk.rbd3.out‚0 disk.rbd4.in‚0Ö3刂(:… €ÆäÔýÔ–x gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±!Øø‚‰‚(:÷ €ÆäÔ›±Ç| disk.dm-0.in‚ 4.9765625 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 253.54296875 disk.rbd1.out‚ 2.20947265625 disk.nvme0n1.in‚ 4.9765625 disk.drbd1777.out‚0 disk.rbd3.in‚0$ memory_percent‚98.94282817840576( disk./.usagePercent‚73.7132134581809& disk.nvme0n1.out‚19191.11669921875 disk.rbd4.out‚0 cpu‚25.253291420386866 network.sent‚ 1.0265828e+07 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda3.out‚ 253.50390625# disk.dm-0.out‚19191.11669921875 disk.dm-1.out‚ 253.50390625 disk.rbd2.in‚0 disk.sda1.out‚0! disk.rbd2.out‚154.22802734375 disk.rbd0.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 173.20703125 disk.rbd1.in‚0* proc.memory.percent‚17.399239540100098 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625( disk.nvme0c0n1.out‚19191.11669921875 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125# proc.memory.rssMB‚ 2850.69140625 network.recv‚ 1.563887e+06% disk./.usageGB‚1098.3801498413086IÑ[b2Š- _wandbruntime‚4345  _runtime‚4345dBìàÇ‹Á & lr-AdamW/pg2‚0.0006298057725181808 trainer/global_step‚4549 % _timestamp‚1.7884208747354126e+09 _runtime‚4345.485354889 _step‚369 & lr-AdamW/pg1‚0.0006298057725181808±€{ÌŒÆ _step‚369 & lr-AdamW/pg1‚0.0006298057725181808 & lr-AdamW/pg2‚0.0006298057725181808 trainer/global_step‚4549 % _timestamp‚1.7884208747354126e+09 _runtime‚4345.485354889ñ²1x2- _wandbruntime‚4345  _runtime‚4345ÃÎýß'ŽÙO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8436232119808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187831305725440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8436232119808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚641 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 _runtime‚4345.486270234 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187831305725440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178279468291530 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118145584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201104096045568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚643 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48820718 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚178279227603022 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169867711132672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8411533509710 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79242113 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201103809575424 % _timestamp‚1.7884208747361274e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83456509 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169867694093312 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8411757158858 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _step‚370 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118119914 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚179395301809152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201602093 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83456507 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30421397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8436003916288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8436003916288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187831551931904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30422038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118145584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 262530048 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 trainer/global_step‚4549 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚179395301809152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83456509 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚643 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48820718 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚178279227603022 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169867711132672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8411533509710 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79242113 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201103809575424 % _timestamp‚1.7884208747361274e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83456509 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169867694093312 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8411757158858 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _step‚370 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118119914 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚179395301809152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201602093 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83456507 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30421397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8436003916288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8436003916288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187831551931904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30422038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118145584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 262530048 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 trainer/global_step‚4549 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚179395301809152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83456509 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187831305725440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8436232119808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚641 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 _runtime‚4345.486270234 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187831305725440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178279468291530 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118145584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201104096045568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ò=$b¨2- _wandbruntime‚4346  _runtime‚4346ø‚Š* ‘ãM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179433639753216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8437841610752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201619691 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201645389 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79259248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚178317362460478 _step‚371 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83474226 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118145465 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30428817 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192559237851648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179433657756160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 % _timestamp‚1.7884208755917428e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83474228 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79259890 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179433639753216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚178317603149194 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169903996551168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8413365909310 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187871727583744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48831073 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48831071 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30428177 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201147161024000 _runtime‚4346.342308081 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83474228 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118171161 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83474226 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚187871727583744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 201619691 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8413589558666 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201645389 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118171161 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187871481363968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8437841610752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8438069827584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192559213911552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8587648072704 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169904013590528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201645389 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118171161 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187871481363968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8437841610752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8438069827584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192559213911552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8587648072704 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169904013590528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201645389 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79259248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚178317362460478 _step‚371 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83474226 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118145465 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30428817 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192559237851648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179433657756160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 % _timestamp‚1.7884208755917428e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83474228 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79259890 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179433639753216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚178317603149194 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169903996551168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8413365909310 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187871727583744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48831073 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48831071 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30428177 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201147161024000 _runtime‚4346.342308081 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83474228 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118171161 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83474226 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561óISûâ2“- _wandbruntime‚4346  _runtime‚4346ÀÍtV”Ð .loss/FRt4/assignment_loss‚3.62575292343996 2loss/SRqqt1/assignment_loss‚1.7579420597425528 2loss/SRqqt2/assignment_loss‚1.4116055518647448 2loss/SRqqt3/assignment_loss‚1.8068870581962444 0loss/FBt1/assignment_loss‚0.2921767469984347 0loss/FBt4/assignment_loss‚0.4360126607902607 1loss/SRqqt4/detection_loss‚1.9414997484002794 /loss/FBt2/detection_loss‚0.7304407678654089 /loss/FRt2/detection_loss‚1.7006891240179538 /loss/FRt4/detection_loss‚2.1831136826349766 1loss/SRqqt1/detection_loss‚2.4518837928771973 %loss/total_loss‚46.14418993089218 _step‚372 0loss/SRqqt4/assignment_loss‚2.02601525879332 -loss/FRt3/assignment_loss‚3.6341975444072 1loss/FBt2/assignment_loss‚0.17483382720572094 /loss/FRt1/detection_loss‚2.3312017314655837 .loss/FRt3/detection_loss‚2.690877764611631 1loss/SRqqt3/detection_loss‚1.8877599661548932 _runtime‚4346.343368699 % _timestamp‚1.7884208755935435e+09 0loss/FRt2/assignment_loss‚2.0364099562168123 0loss/FBt3/assignment_loss‚0.5783374619416215 1loss/SRqqt2/detection_loss‚1.2566663851892506 /loss/FBt4/detection_loss‚2.7414436075422497 epoch‚3 /loss/FRt1/assignment_loss‚3.655550618504369 /loss/FBt1/detection_loss‚2.6044786843386563 .loss/FBt3/detection_loss‚2.188413007692857 trainer/global_step‚4549¢Ø‡Û[•Õ % _timestamp‚1.7884208755935435e+09 0loss/FRt2/assignment_loss‚2.0364099562168123 0loss/FBt3/assignment_loss‚0.5783374619416215 1loss/SRqqt2/detection_loss‚1.2566663851892506 /loss/FBt4/detection_loss‚2.7414436075422497 epoch‚3 /loss/FRt1/assignment_loss‚3.655550618504369 /loss/FBt1/detection_loss‚2.6044786843386563 .loss/FBt3/detection_loss‚2.188413007692857 trainer/global_step‚4549 .loss/FRt4/assignment_loss‚3.62575292343996 2loss/SRqqt1/assignment_loss‚1.7579420597425528 2loss/SRqqt2/assignment_loss‚1.4116055518647448 2loss/SRqqt3/assignment_loss‚1.8068870581962444 0loss/FBt1/assignment_loss‚0.2921767469984347 0loss/FBt4/assignment_loss‚0.4360126607902607 1loss/SRqqt4/detection_loss‚1.9414997484002794 /loss/FBt2/detection_loss‚0.7304407678654089 /loss/FRt2/detection_loss‚1.7006891240179538 /loss/FRt4/detection_loss‚2.1831136826349766 1loss/SRqqt1/detection_loss‚2.4518837928771973 %loss/total_loss‚46.14418993089218 _step‚372 0loss/SRqqt4/assignment_loss‚2.02601525879332 -loss/FRt3/assignment_loss‚3.6341975444072 1loss/FBt2/assignment_loss‚0.17483382720572094 /loss/FRt1/detection_loss‚2.3312017314655837 .loss/FRt3/detection_loss‚2.690877764611631 1loss/SRqqt3/detection_loss‚1.8877599661548932 _runtime‚4346.343368699ôÔrš–‚(: ÆäÔ£˜¦x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.751' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.750200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 íÑ›~—‚(:ó ÆäÔ¸º€{ disk.dm-1.out‚ 254.1953125$ memory_percent‚98.94430637359619 disk.sda.out‚ 254.234375 disk.nvme0n1.in‚ 4.9765625 disk.dm-0.in‚ 4.9765625 network.sent‚ 1.0271796e+07 disk.drbd1777.out‚0 disk.sda3.in‚7.3125 disk.rbd1.in‚0& disk.nvme0n1.out‚19290.19384765625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda3.out‚ 254.1953125 disk.rbd1.out‚ 2.20947265625# proc.memory.rssMB‚ 2850.69140625* proc.memory.percent‚17.399239540100098( disk.nvme0c0n1.out‚19290.19384765625 network.recv‚ 1.568228e+06% disk./.usageGB‚1098.3801765441895 disk.nvme0c0n1.in‚ 4.9765625 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125# disk.dm-0.out‚19290.19384765625( proc.memory.availableMB‚ 172.96484375 cpu‚25.25261310275588) disk./.usagePercent‚73.71321525023365 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda2.in‚0! disk.rbd2.out‚155.86474609375 disk.rbd3.out‚0 disk.dm-1.in‚7.3125ºÔˆ˜‚(:… žÆäÔ´¯šx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d¸ü´€™‚(:õ žÆäÔÑ®Ïz$ memory_percent‚98.94435405731201# proc.memory.rssMB‚ 2850.69140625 cpu‚25.25192567401836 network.recv‚ 1.574282e+06 disk.sda1.in‚ 0.05078125# disk.dm-0.out‚19389.61474609375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚19389.61474609375 disk.sda1.out‚0 disk.sda.in‚ 7.36328125( proc.memory.availableMB‚ 172.95703125$ disk./.usageGB‚1098.379981994629 disk.dm-0.in‚ 4.9765625 disk.dm-1.out‚ 254.81640625 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 proc.cpu.threads‚17 network.sent‚ 1.0380179e+07( disk./.usagePercent‚73.7132021938494 disk.rbd1.out‚ 2.20947265625 disk.dm-1.in‚7.3125 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd3.in‚0 disk.nvme0n1.in‚ 4.9765625! disk.rbd2.out‚155.88037109375 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 254.85546875 disk.sda3.in‚7.3125* proc.memory.percent‚17.399239540100098 disk.sda3.out‚ 254.81640625 disk.rbd1.in‚0& disk.nvme0n1.out‚19389.61474609375 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625w}š‚(:… ­ÆäÔÖ´Êx gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.011' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a/Âz›‚(:ï ­ÆäÔ¿™Ýz network.sent‚ 1.0386497e+07 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.drbd1777.in‚ 1.03515625& proc.memory.availableMB‚ 172.921875 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 cpu‚25.251236970360527 proc.cpu.threads‚17 disk.rbd2.in‚0 network.recv‚ 1.578729e+06& disk.nvme0c0n1.out‚19488.806640625 disk.sda3.out‚ 255.34765625 disk.rbd1.in‚0 disk.rbd1.out‚ 2.20947265625 disk.dm-0.in‚ 4.9765625 disk.sda2.in‚0 disk.sda1.in‚ 0.05078125$ disk.nvme0n1.out‚19488.806640625 disk.dm-1.out‚ 255.34765625# proc.memory.rssMB‚ 2850.69140625* proc.memory.percent‚17.399239540100098) disk./.usagePercent‚73.71317838514871 disk.rbd4.out‚0 disk.sda3.in‚7.3125 disk.dm-1.in‚7.3125 disk.sda.out‚ 255.38671875 disk.rbd0.out‚0 disk.drbd1777.out‚0# memory_percent‚98.9445686340332! disk.rbd2.out‚155.94287109375% disk./.usageGB‚1098.3796272277832 disk.nvme0n1.in‚ 4.9765625! disk.dm-0.out‚19488.806640625 disk.sda2.out‚ 0.0390625R7I)2œ- _wandbruntime‚4390  _runtime‚4390£‹ÁÇÁ & lr-AdamW/pg1‚0.0006250236667218877 & lr-AdamW/pg2‚0.0006250236667218877 trainer/global_step‚4599 % _timestamp‚1.7884209197870214e+09 _runtime‚4390.537005431 _step‚373$\i·ÌžÆ _step‚373 & lr-AdamW/pg1‚0.0006250236667218877 & lr-AdamW/pg2‚0.0006250236667218877 trainer/global_step‚4599 % _timestamp‚1.7884209197870214e+09 _runtime‚4390.537005431õ&@Ðc2Ÿ- _wandbruntime‚4390  _runtime‚4390_~ˆß' ÙO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚653 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119400816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203743273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84342457 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189840102065664 % _timestamp‚1.7884209197877805e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8503167080918 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84342459 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚181312199012352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180185984067030 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189840348272128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚171682816986112 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49338468 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚4390.538002559 _step‚374 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8503390730066 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119426486 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194576552968192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚181312217015296 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181312199012352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚203255723232256 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8528131256832 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119426486 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194576552968192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚181312217015296 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181312199012352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚203255723232256 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8528131256832 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189840102065664 % _timestamp‚1.7884209197877805e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8503167080918 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84342459 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚181312199012352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180185984067030 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189840348272128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚171682816986112 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49338468 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚4390.538002559 _step‚374 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8503390730066 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203786543 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30753474 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8529740747776 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84360176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189880523923968 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203298752654336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8681333223936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80102959 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49348823 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚181350536956416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84360176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚189880523923968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203786543 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 _runtime‚4391.399029144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8529740747776 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚375 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8504999480518 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8505223129874 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119452063 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8529968964608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194617722664448 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203812241 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚189880277704192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84360178 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80102295 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180224118924486 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49348821 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8529968964608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 % _timestamp‚1.7884209206483924e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119426367 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194617698724352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203812241 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84360178 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171719119443968 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262¤Ø"Áî&¤èM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84360178 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171719119443968 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚181350536956416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 trainer/global_step‚4599 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203786543 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30753474 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8529740747776 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84360176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189880523923968 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203298752654336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8681333223936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80102959 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49348823 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚181350536956416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84360176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚189880523923968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203786543 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 _runtime‚4391.399029144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8529740747776 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚375 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8504999480518 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8505223129874 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119452063 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8529968964608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194617722664448 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203812241 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚189880277704192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84360178 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80102295 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180224118924486 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49348821 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8529968964608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 % _timestamp‚1.7884209206483924e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119426367 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194617698724352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203812241 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624÷I'g2¥- _wandbruntime‚4391  _runtime‚4391-ÊÿUU¦Ï /loss/FRt4/assignment_loss‚3.647578493754069 1loss/FBt4/assignment_loss‚0.21404889719477588 % _timestamp‚1.7884209206502953e+09 /loss/FRt1/assignment_loss‚4.300711997917721 /loss/FRt3/assignment_loss‚3.819697232473464 /loss/FBt1/assignment_loss‚0.431831174663135 _step‚376 1loss/SRqqt3/assignment_loss‚1.880479837588051 1loss/FBt3/assignment_loss‚0.06243780514341779 1loss/SRqqt3/detection_loss‚1.6885521521693783 1loss/SRqqt4/detection_loss‚2.1164605153931513 /loss/FBt4/detection_loss‚2.8181266444069997 .loss/FRt4/detection_loss‚2.740206968784334 -loss/SRqqt1/detection_loss‚3.136962890625 /loss/FBt1/detection_loss‚2.3738718245710646 epoch‚3 trainer/global_step‚4599 _runtime‚4391.400109069 2loss/SRqqt4/assignment_loss‚1.7438403273622194 1loss/FBt2/assignment_loss‚0.17388816541080235 .loss/FRt1/detection_loss‚2.329556731950669 /loss/FBt2/detection_loss‚0.6262429966250571 /loss/FBt3/detection_loss‚2.9208792224526405 2loss/SRqqt1/assignment_loss‚1.9259777189720244 /loss/FRt3/detection_loss‚2.3491864271816754 .loss/FRt2/detection_loss‚2.098026588204361 0loss/FRt2/assignment_loss‚1.9889594444206782 2loss/SRqqt2/assignment_loss‚1.3078003510973364 0loss/SRqqt2/detection_loss‚1.219126765036358 %loss/total_loss‚47.91445117339838Eƒ&vZ§Ô _step‚376 1loss/SRqqt3/assignment_loss‚1.880479837588051 1loss/FBt3/assignment_loss‚0.06243780514341779 1loss/SRqqt3/detection_loss‚1.6885521521693783 1loss/SRqqt4/detection_loss‚2.1164605153931513 /loss/FBt4/detection_loss‚2.8181266444069997 .loss/FRt4/detection_loss‚2.740206968784334 -loss/SRqqt1/detection_loss‚3.136962890625 /loss/FBt1/detection_loss‚2.3738718245710646 epoch‚3 trainer/global_step‚4599 _runtime‚4391.400109069 2loss/SRqqt4/assignment_loss‚1.7438403273622194 1loss/FBt2/assignment_loss‚0.17388816541080235 .loss/FRt1/detection_loss‚2.329556731950669 /loss/FBt2/detection_loss‚0.6262429966250571 /loss/FBt3/detection_loss‚2.9208792224526405 2loss/SRqqt1/assignment_loss‚1.9259777189720244 /loss/FRt3/detection_loss‚2.3491864271816754 .loss/FRt2/detection_loss‚2.098026588204361 0loss/FRt2/assignment_loss‚1.9889594444206782 2loss/SRqqt2/assignment_loss‚1.3078003510973364 0loss/SRqqt2/detection_loss‚1.219126765036358 %loss/total_loss‚47.91445117339838 /loss/FRt4/assignment_loss‚3.647578493754069 1loss/FBt4/assignment_loss‚0.21404889719477588 % _timestamp‚1.7884209206502953e+09 /loss/FRt1/assignment_loss‚4.300711997917721 /loss/FRt3/assignment_loss‚3.819697232473464 /loss/FBt1/assignment_loss‚0.431831174663135øèža䛨‚(: ¼ÆäÔÞ£«x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØæP»~©‚(:ó ¼ÆäÔ¦Áîz disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda3.out‚ 256.0703125& disk.nvme0n1.out‚19490.34033203125( disk.nvme0c0n1.out‚19490.34033203125$ memory_percent‚98.94444942474365 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 2.20947265625# disk.dm-0.out‚19490.34033203125 disk.drbd1777.in‚ 1.03515625* proc.memory.percent‚17.399239540100098 network.sent‚ 1.0391759e+07) disk./.usagePercent‚73.69572865560737$ disk./.usageGB‚1098.119613647461 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2850.69140625 disk.sda.out‚ 256.109375 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.nvme0n1.in‚ 4.9765625 disk.dm-1.out‚ 256.0703125! disk.rbd2.out‚155.95849609375 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.dm-0.in‚ 4.9765625 cpu‚25.250496093002656 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 4.9765625 proc.cpu.threads‚17 network.recv‚ 1.581685e+06 disk.rbd1.in‚0( proc.memory.availableMB‚ 172.94140625Fí š›ª‚(: ËÆäÔ«²´x gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.436' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.887200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¼Ú"«‚(:ñ ËÆäÔ˜¯êz disk.rbd3.out‚0 disk.sda.in‚ 7.36328125# disk.dm-0.out‚19755.65283203125( proc.memory.availableMB‚ 172.91796875# proc.memory.rssMB‚ 2850.69140625 cpu‚25.249818595047305 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.sda1.in‚ 0.05078125& disk.nvme0n1.out‚19755.65283203125) disk./.usagePercent‚73.64191612791758 disk.dm-1.out‚ 256.765625 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd1.out‚ 2.20947265625* proc.memory.pe2ù؈Zrcent‚17.399239540100098 proc.cpu.threads‚17 disk.rbd2.out‚ 156.115234375 disk.rbd4.in‚0( disk.nvme0c0n1.out‚19755.65283203125 network.recv‚ 1.587951e+06 disk.sda3.out‚ 256.765625 disk.sda2.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 4.9765625 disk.rbd0.in‚0 disk.sda.out‚ 256.8046875$ memory_percent‚98.94459247589111 disk.drbd1777.out‚0 network.sent‚ 1.0499349e+07% disk./.usageGB‚1097.3177680969238 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.sda3.in‚7.3125 disk.rbd1.in‚0 disk.dm-0.in‚ 4.9765625 disk.sda1.out‚0ûGc¬‚(:„ ÚÆäÔ½á‚x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.235' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.647 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡«W7v­‚(:ë ÚÆäÔÃôŒ|$ memory_percent‚98.94311428070068" proc.memory.rssMB‚ 2850.6953125) disk./.usagePercent‚73.64190793567649$ disk.nvme0n1.out‚19756.333984375! disk.dm-0.out‚19756.333984375 disk.rbd0.in‚0( proc.memory.availableMB‚ 173.16015625 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 2.20947265625 network.recv‚ 1.591867e+06 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 4.9765625 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 257.4765625 disk.nvme0n1.in‚ 4.9765625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.9765625 cpu‚25.249130410719005% disk./.usageGB‚1097.3176460266113 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.sda2.out‚ 0.0390625* proc.memory.percent‚17.399263381958008 disk.sda.out‚ 257.515625 disk.sda3.in‚7.3125& disk.nvme0c0n1.out‚19756.333984375 disk.drbd1777.in‚ 1.03515625 network.sent‚ 1.0505261e+07 disk.sda1.in‚ 0.05078125 disk.rbd4.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 257.4765625 disk.rbd2.out‚ 156.115234375 disk.dm-1.in‚7.3125 disk.rbd4.in‚0Wý6¨2®- _wandbruntime‚4436  _runtime‚4436×tTƯÀ & lr-AdamW/pg1‚0.0006201949079345096 & lr-AdamW/pg2‚0.0006201949079345096 trainer/global_step‚4649 $ _timestamp‚1.788420965394988e+09 _runtime‚4436.144973942 _step‚377|Pö"Ë°Å trainer/global_step‚4649 $ _timestamp‚1.788420965394988e+09 _runtime‚4436.144973942 _step‚377 & lr-AdamW/pg1‚0.0006201949079345096 & lr-AdamW/pg2‚0.0006201949079345096ùÏ‹2±- _wandbruntime‚4436  _runtime‚4436k±ß'²ÙO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8594800652126 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120707388 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313733120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚183229114218496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚191849144612352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80931383 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 % _timestamp‚1.7884209653957164e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205935797 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _runtime‚4436.145885029 _step‚378 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 trainer/global_step‚4649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182092981219546 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85228407 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚196635037780992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8620030393856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31074485 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205407931527680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚183229096215552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80930701 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191848898405888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205407617794560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173497939878912 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120707388 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49856218 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182092740531038 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8619802190336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8595024301274 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191848898405888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85228407Ô–ßíä'³ÞO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚183229114218496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚191849144612352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80931383 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 % _timestamp‚1.7884209653957164e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205935797 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _runtime‚4436.145885029 _step‚378 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 trainer/global_step‚4649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182092981219546 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85228407 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚196635037780992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8620030393856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31074485 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205407931527680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚183229096215552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80930701 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191848898405888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205407617794560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173497939878912 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120707388 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49856218 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182092740531038 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8619802190336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8595024301274 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191848898405888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85228407 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8594800652126 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120707388 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313733120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0úƒÏF2´- _wandbruntime‚4437  _runtime‚44378`l«j%µãM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205953395 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31081508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31080896 _runtime‚4437.004934173 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120707269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80947467 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49866573 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120732965 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182131116077210 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8621868101632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205450952990208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205979093 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191889320264192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8621639884800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49866571 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8774476704768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173534259376128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173534242336768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80948081 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205450660241920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8621868101632 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120707269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚196676183537152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205979093 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85246126 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 cODeviceStatsMonitor.on_train_batch_end/inaÚ–*ctive_split_bytes.large_pool.allocated‚196676207477248 _step‚379 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191889320264192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299Ž¾cî&¶èM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚183267434159616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚183267452162560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8596633051726 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85246126 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85246126 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚196676207477248 _step‚379 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191889320264192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205953395 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31081508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31080896 _runtime‚4437.004934173 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120707269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80947467 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49866573 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120732965 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182131116077210 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8621868101632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205450952990208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205979093 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191889320264192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8621639884800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49866571 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8774476704768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173534259376128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173534242336768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80948081 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205450660241920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8621868101632 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120707269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚196676183537152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205979093 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8711929530880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198693498653696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8866380823040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 % _timestamp‚1.7884210109780984e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121988290 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8711701327360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121962620 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8866109904384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193857694746112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚185145993418752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚207559903416832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81775527 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8686657872482 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚175313062771712 trainer/global_step‚4699 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86114357 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚185145993418752 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86114359 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚207559608558080 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208076977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193857940952576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81776202 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183999496995046 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50373968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121988290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193857940952576 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31401561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8711929530880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208102649 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚175313079811072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8686434223334 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 _runtime‚ 4481.72823401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50373966 _step‚382 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31401561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8711929530880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208102649 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚175313079811072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8686434223334 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 _runtime‚ 4481.72823401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50373966 _step‚382 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8711929530880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198693498653696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8866380823040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 % _timestamp‚1.7884210109780984e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121988290 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8711701327360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121962620 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8866109904384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193857694746112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚185145993418752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚207559903416832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81775527 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8686657872482þÛ(+2Æ- _wandbruntime‚4482  _runtime‚4482¡µé&ÇãM _runtime‚4482.591291714 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚185184349365760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193897870384640 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚185184349365760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81792182 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198734668349952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86132076 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚207602942475776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8868250185728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚185184331362816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50384323 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198734692290048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121988171 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8713767238656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8713539021824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208145945 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚175349382268928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8688266622934 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31407861 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚184037872541218 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 % _timestamp‚1.7884210118406487e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86132078 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8713767238656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208120247 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8688490272290 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193898116604416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121988171 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8713539021824 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚207602643436032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86132076 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50384321 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653†úíî&ÈèM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86132076 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚207602942475776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8868250185728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚185184331362816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50384323 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198734692290048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121988171 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8713767238656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8713539021824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208145945 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚175349382268928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8688266622934 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31407861 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚184037872541218 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 % _timestamp‚1.7884210118406487e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86132078 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8713767238656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208120247 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8688490272290 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193898116604416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121988171 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8713539021824 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚207602643436032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86132076 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50384321 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 _runtime‚4482.591291714 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚185184349365760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193897870384640 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚185184349365760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81792182 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198734668349952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673ÿ>„¦£2É- _wandbruntime‚4482  _runtime‚4482*¯YWÊÑ 1loss/SRqqt2/assignment_loss‚1.428267956635014 epoch‚3 0loss/FRt4/assignment_loss‚3.7799774510519843 1loss/SRqqt4/detection_loss‚1.8189605579332069 _runtime‚4482.592294323 0loss/FRt3/assignment_loss‚3.5572044026851652 .loss/FBt3/detection_loss‚3.105606012046337 .loss/FBt4/detection_loss‚2.402378953420199 .loss/FRt1/assignment_loss‚3.84559674884962 2loss/SRqqt4/assignment_loss‚2.0780157046737493 2loss/FBt3/assignment_loss‚0.006083205596951302 2loss/FBt4/assignment_loss‚0.025039149582600936 .loss/FRt3/detection_loss‚2.201157583594322 % _timestamp‚1.7884210118424911e+09 -loss/FRt4/detection_loss‚2.38258181469783 1loss/SRqqt3/detection_loss‚1.5328466368132625 /loss/FBt2/detection_loss‚0.5878332211421087 1loss/SRqqt1/detection_loss‚2.4157665931817256 /loss/FBt1/detection_loss‚3.4709490537643433 trainer/global_step‚4699 2loss/SRqqt3/assignment_loss‚1.8096776064806457 1loss/FBt1/assignment_loss‚0.42692211577668787 1loss/FBt2/assignment_loss‚0.19754896543598094 .loss/FRt1/detection_loss‚2.526776995509863 .loss/FRt2/detection_loss‚1.691298323742887 _step‚384 .loss/FRt2/assignment_loss‚1.86151178846968 0loss/SRqqt1/assignment_loss‚2.11219162922917 1loss/SRqqt2/detection_loss‚1.1018523652045453 %loss/total_loss‚46.36604483551788¼âm\ËÖ -loss/FRt4/detection_loss‚2.38258181469783 1loss/SRqqt3/detection_loss‚1.5328466368132625 /loss/FBt2/detection_loss‚0.5878332211421087 1loss/SRqqt1/detection_loss‚2.4157665931817256 /loss/FBt1/detection_loss‚3.4709490537643433 trainer/global_step‚4699 2loss/SRqqt3/assignment_loss‚1.8096776064806457 1loss/FBt1/assignment_loss‚0.42692211577668787 1loss/FBt2/assignment_loss‚0.19754896543598094 .loss/FRt1/detection_loss‚2.526776995509863 .loss/FRt2/detection_loss‚1.691298323742887 _step‚384 .loss/FRt2/assignment_loss‚1.86151178846968 0loss/SRqqt1/assignment_loss‚2.11219162922917 1loss/SRqqt2/detection_loss‚1.1018523652045453 %loss/total_loss‚46.36604483551788 1loss/SRqqt2/assignment_loss‚1.428267956635014 epoch‚3 0loss/FRt4/assignment_loss‚3.7799774510519843 1loss/SRqqt4/detection_loss‚1.8189605579332069 _runtime‚4482.592294323 0loss/FRt3/assignment_loss‚3.5572044026851652 .loss/FBt3/detection_loss‚3.105606012046337 .loss/FBt4/detection_loss‚2.402378953420199 .loss/FRt1/assignment_loss‚3.84559674884962 2loss/SRqqt4/assignment_loss‚2.0780157046737493 2loss/FBt3/assignment_loss‚0.006083205596951302 2loss/FBt4/assignment_loss‚0.025039149582600936 .loss/FRt3/detection_loss‚2.201157583594322 % _timestamp‚1.7884210118424911e+09€“ÞÎÌ‚(:… –ÇäÔ°è´x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.896' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æHp|Í‚(:ñ –ÇäÔ܉ñz disk.sda1.in‚ 0.05078125 proc.cpu.threads‚17) disk./.usagePercent‚73.64193072034703 disk.dm-1.in‚7.3125" proc.memory.rssMB‚ 2850.6953125 network.sent‚ 1.0631365e+07 disk.rbd2.in‚0! disk.rbd2.out‚157.80322265625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.sda2.out‚ 0.0390625( proc.memory.availableMB‚ 173.31640625 disk.dm-1.out‚ 259.6640625 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd3.out‚0$ disk./.usageGB‚1097.317985534668 disk.drbd1777.out‚0 disk.rbd4.out‚0& disk.nvme0n1.out‚19758.39306640625 disk.dm-0.in‚ 4.9765625 disk.sda2.in‚0 disk.rbd4.in‚0( disk.nvme0c0n1.out‚19758.39306640625 disk.sda3.out‚ 259.6640625 disk.nvme0n1.in‚ 4.9765625* proc.memory.percent‚17.399263381958008 disk.rbd1.in‚0# disk.dm-0.out‚19758.39306640625 network.recv‚ 1.610898e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 259.703125 disk.rbd1.out‚ 2.20947265625$ memory_percent‚98.94216060638428 cpu‚25.24649610781927 disk.drbd1777.in‚ 1.03515625´k½Î‚(:… ¥ÇäÔåì˜x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚100.063' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.0126 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]© Ï‚(:ô ¥ÇäÔò™çz disk.rbd0.in‚0 disk.sda.out‚ 260.24609375 disk.sda2.in‚0! disk.rbd2.out‚157.96728515625 disk.rbd4.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.9765625# disk.dm-0.out‚19758.71337890625 disk.drbd1777.in‚ 1.03515625$ memory_percent‚98.94552230834961& proc.memory.availableMB‚ 172.765625 cpu‚25.245790633474613 network.recv‚ 1.617707e+06 disk.sda3.in‚7.3125 disk.rbd1.out‚ 2.20947265625 disk.rbd2.in‚0" proc.memory.rssMB‚ 2850.6953125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚19758.71337890625 disk.drbd1777.out‚0* proc.memory.percent‚17.399263381958008 disk.sda.in‚ 7.36328125% disk./.usageGB‚1097.3177299499512 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda3.out‚ 260.20703125 disk.dm-0.in‚ 4.9765625 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 network.sent‚ 1.074019e+07 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 260.20703125) disk./.usagePercent‚73.64191356784224 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd3.in‚0& disk.nvme0n1.out‚19758.71337890625Úo›Ð‚(: ´ÇäÔ¡¿Ÿx gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.727' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.945400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m_ñzQÑ‚(:ö ´ÇäÔ®Øz network.recv‚ 1.621506e+06# disk.dm-0.out‚19759.24951171875 disk.dm-1.out‚ 260.90234375 disk.rbd0.out‚0 disk.sda.out‚ 260.94140625 disk.rbd1.out‚ 2.20947265625 disk.nvme0n1.in‚ 4.9765625$ memory_percent‚98.94554615020752* proc.memory.percent‚17.399263381958008 network.senâ¬0t‚ 1.0745513e+07 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd2.in‚0 cpu‚25.24514540229851! disk.rbd2.out‚159.47900390625( disk.nvme0c0n1.out‚19759.24951171875 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625) disk./.usagePercent‚73.64191945601553% disk./.usageGB‚1097.3178176879883 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0" proc.memory.rssMB‚ 2850.6953125 disk.rbd3.out‚0 disk.sda3.out‚ 260.90234375& disk.nvme0n1.out‚19759.24951171875 disk.dm-0.in‚ 4.9765625 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 172.76171875 proc.cpu.threads‚17ψ´¦2Ò- _wandbruntime‚4527  _runtime‚4527HK]TÇÓÁ % _timestamp‚1.7884210564645896e+09 _runtime‚4527.214520153 _step‚385 & lr-AdamW/pg1‚0.0006104014661235751 & lr-AdamW/pg2‚0.0006104014661235751 trainer/global_step‚4749*C®ÌÔÆ & lr-AdamW/pg1‚0.0006104014661235751 & lr-AdamW/pg2‚0.0006104014661235751 trainer/global_step‚4749 % _timestamp‚1.7884210564645896e+09 _runtime‚4527.214520153 _step‚3854IÞ¼2Õ- _wandbruntime‚4527  _runtime‚4527Hµ.Þ'ÖØO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87000309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87000307 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200751983466496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210243829 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚187062908624896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚195866737292800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210269501 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4749 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚195866491086336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50891716 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31728607 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8959804486144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 _step‚386 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187062890621952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚185906494147562 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195866491086336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8803600464384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31727909 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209711510742528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185906253459054 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177128185664512 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187062908624896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8803828667904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123269192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200752007406592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884210564653888e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚195866737292800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210269501 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4749 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚195866491086336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50891716 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31728607 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8959804486144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 _step‚386 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187062890621952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚185906494147562 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195866491086336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8803600464384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31727909 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209711510742528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185906253459054 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 296082944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177128185664512 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187062908624896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8803828667904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123269192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200752007406592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884210564653888e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚187062908624896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123294769 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82636216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50902073 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209754845948928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚195906912944640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8779900194142 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187101246568960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82636848 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31734775 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚177164505161728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87018028 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50902071 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187101246568960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187101228566016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚200793153162752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87018026 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210312797 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210287099 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 _step‚387 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185944629005226 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210312797 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚195906666724864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87018028 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8961393746432Îç7¢·ÚèM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87018028 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 296082944 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50902071 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187101246568960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187101228566016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚200793153162752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87018026 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210312797 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210287099 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 _step‚387 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185944629005226 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210312797 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚195906666724864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87018028 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8961393746432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 trainer/global_step‚4749 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 % _timestamp‚1.7884210573432407e+09 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123294769 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82636216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50902073 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209754845948928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚195906912944640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8779900194142 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187101246568960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82636848 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31734775 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚177164505161728ƒr§+d2Û- _wandbruntime‚4528  _runtime‚4528ä}c]Ü× 1loss/FBt3/assignment_loss‚0.38196452202585834 0loss/FBt4/assignment_loss‚0.2357032617498286 1loss/SRqqt1/detection_loss‚2.0944948763382145 1loss/SRqqt4/detection_loss‚2.3680222138415936 trainer/global_step‚4749 _step‚388 0loss/FRt1/assignment_loss‚3.7058509409427645 0loss/FRt3/assignment_loss‚4.0504764881398945 /loss/FRt4/detection_loss‚2.6403669973214465 /loss/FBt1/detection_loss‚2.2335149115986295 % _timestamp‚1.7884210573453903e+09 .loss/FRt1/detection_loss‚2.541984051093459 0loss/FRt2/assignment_loss‚2.0353593188419676 1loss/SRqqt1/assignment_loss‚2.048599906447457 /loss/FRt3/detection_loss‚2.6306915572947926 /loss/FBt2/detection_loss‚0.4951042790861824 /loss/FBt4/detection_loss‚2.6924068133036294 epoch‚3 /loss/FBt3/detection_loss‚2.6996267239252725 0loss/FBt2/assignment_loss‚0.1775832764364731 /loss/FRt4/assignment_loss‚3.938102428118388 2loss/SRqqt3/assignment_loss‚1.8703084686335096 /loss/FRt2/detection_loss‚1.5745918562537746 1loss/SRqqt3/detection_loss‚1.8476091077987185 %loss/total_loss‚47.03155825325902 2loss/SRqqt2/assignment_loss‚1.2620069318647056 2loss/SRqqt4/assignment_loss‚2.0192776407266773 0loss/FBt1/assignment_loss‚0.5881161871883605 1loss/SRqqt2/detection_loss‚0.8997954942874218 _runtime‚4528.095218123ÂõbÝÜ 2loss/SRqqt3/assignment_loss‚1.8703084686335096 /loss/FRt2/detection_loss‚1.5745918562537746 1loss/SRqqt3/detection_loss‚1.8476091077987185 %loss/total_loss‚47.03155825325902 2loss/SRqqt2/assignment_loss‚1.2620069318647056 2loss/SRqqt4/assignment_loss‚2.0192776407266773 0loss/FBt1/assignment_loss‚0.5881161871883605 1loss/SRqqt2/detection_loss‚0.8997954942874218 _runtime‚4528.095218123 1loss/FBt3/assignment_loss‚0.38196452202585834 0loss/FBt4/assignment_loss‚0.2357032617498286 1loss/SRqqt1/detection_loss‚2.0944948763382145 1loss/SRqqt4/detection_loss‚2.3680222138415936 trainer/global_step‚4749 _step‚388 0loss/FRt1/assignment_loss‚3.7058509409427645 0loss/FRt3/assignment_loss‚4.0504764881398945 /loss/FRt4/detection_loss‚2.6403669973214465 /loss/FBt1/detection_loss‚2.2335149115986295 % _timestamp‚1.7884210573453903e+09 .loss/FRt1/detection_loss‚2.541984051093459 0loss/FRt2/assignment_loss‚2.0353593188419676 1loss/SRqqt1/assignment_loss‚2.048599906447457 /loss/FRt3/detection_loss‚2.6306915572947926 /loss/FBt2/detection_loss‚0.4951042790861824 /loss/FBt4/detection_loss‚2.6924068133036294 epoch‚3 /loss/FBt3/detection_loss‚2.6996267239252725 0loss/FBt2/assignment_loss‚0.1775832764364731 /loss/FRt4/assignment_loss‚3.938102428118388„,®ÕÞ‚(:„ ÃÇäÔµº¤x gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.932' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3864 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,Çsxß‚(:í ÃÇäÔŸ«²{% proc.memory.availableMB‚ 172.78125 disk.sda2.out‚ 0.0390625 disk.sda3.in‚7.3125 network.sent‚ 1.0751542e+07 disk.sda2.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.sda.out‚ 261.3671875) disk./.usagePercent‚73.63048180740827& disk.nvme0n1.out‚19761.45654296875! disk.rbd2.out‚159.61962890625 disk.sda1.out‚0$ disk./.usageGB‚1097.147388458252 disk.rbd1.in‚0 disk.rbd1.out‚ 2.20947265625# disk.dm-0.out‚19761.45654296875 disk.dm-1.in‚7.3125* proc.memory.percent‚17.399263381958008 disk.drbd1777.out‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚19761.45654296875 disk.rbd3.out‚0$ memory_percent‚98.94542694091797" proc.memory.rssMB‚ 2850.6953125 disk.nvme0n1.in‚ 4.9765625 disk.rbd4.in‚0 disk.sda.in‚ 7.36328125 disk.dm-1.out‚ 261.328125 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.24449471049438 proc.cpu.threads‚17 network.recv‚ 1.625956e+06 disk.dm-0.in‚ 4.9765625 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚ 4.9765625 disk.sda3.out‚ 261.328125ùFà‚(:… ÒÇäÔ؆´x gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.011' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;“ò’‚á‚(:÷ ÒÇäÔ”©ëz) disk./.usagePercent‚73.63921217634093# disk.dm-0.out‚19896.16943359375 disk.sda2.out‚ 0.0390625( disk.nvme0c0n1.out‚19896.16943359375 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125& disk.nvme0n1.out‚19896.16943359375 disk.rbd0.out‚0 network.sent‚ 1.0859712e+07 network.recv‚ 1.631768e+06 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 2.20947265625 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda.out‚ 261.86328125 disk.rbd1.in‚0 disk.dm-0.in‚ 4.9765625 disk.dm-1.in‚7.3125 disk.nvme0c0n1.in‚ 4.9765625" proc.memory.rssMB‚ 2850.6953125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.out‚ 261.82421875 disk.drbd1777.out‚0* proc.memory.percent‚17.399263381958008 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 cpu‚25.243862955560104% disk./.usageGB‚1097.2774772644043 disk.rbd3.out‚0 disk.nvme0n1.in‚ 4.9765625$ memory_percent‚98.94592761993408 disk.sda3.out‚ 261.82421875! disk.rbd2.out‚159.61962890625 disk.rbd4.out‚0( proc.memory.availableMB‚ 172.69921875ÿéÂŽâ‚(:ƒ áÇäÔФ®x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.365' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.273 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XÖx~ã‚(:ó áÇäÔ•å¥z$ memory_percent‚98.94688129425049 disk.nvme0c0n1.in‚ 4.9765625 network.sent‚ 1.0866052e+07 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.9765625" proc.memory.rssMB‚ 2850.6953125* proc.memory.percent‚17.399263381958008 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.sda.out‚ 262.26953125 disk.sda3.out‚ 262.23046875 disk.sda2.in‚0& disk.nvme0n1.out‚19896.42333984375# disk.dm-0.out‚19896.42333984375 disk.rbd2.out‚159.7998046875 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 172.54296875( disk.nvme0c0n1.out‚19896.42333984375 network.recv‚ 1.63662e+06( disk./.usagePercent‚73.6479545776277 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 2.20947265625 disk.dm-1.in‚7.3125 proc.cpu.threads‚17 disk.sda3.in‚7.3125 disk.dm-0.in‚ 4.9765625 disk.rbd4.out‚0 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 cpu‚25.24323516792298% disk./.usageGB‚1097.4077453613281 disk.dm-1.out‚ 262.23046875Sƒê2ä- _wandbruntime‚4572  _runtime‚4572C<ÃkÅå¿ % _timestamp‚1.7884211021461685e+09 _runtime‚4572.896084439 _step‚389 % lr-AdamW/pg1‚0.000605438824038613 % lr-AdamW/pg2‚0.000605438824038613 trainer/global_step‚4799ötÊæÄ trainer/global_step‚4799 % _timestamp‚1.7884211021461685e+09 _runtime‚4572.896084439 _step‚389 % lr-AdamW/pg1‚0.000605438824038613 % lr-AdamW/pg2‚0.000605438824038613…e$ 2ç- _wandbruntime‚4572  _runtime‚4572j‹>Þ'èØO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32051303 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8895499601408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚187813250611570 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 $ _timestamp‚1.788421102147029e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211863793293312 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 _runtime‚4572.897318163 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚197875533633024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8895727804928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9053034349568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8869701365750 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124550094 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197875287426560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚187813009923062 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87886259 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚197875533633024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32051949 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83460769 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51409466 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124550094 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87886259 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87886257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83461417 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚178943308557312 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8895727804928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212436353 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724352 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212410681 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188979787825152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202810468279296 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124524424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51409468 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202810492219392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚390 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188979805828096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124524424 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚188979787825152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211863502628864 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9053301073920 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51409468 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202810492219392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚390 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188979805828096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124524424 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚188979787825152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211863502628864 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9053301073920 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197875287426560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚187813009923062 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87886259 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚197875533633024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32051949 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83460769 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51409466†7Û6Ì2ê- _wandbruntime‚4573  _runtime‚4573ƒÿRé&ëãM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚189018125769216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87903978 _runtime‚4573.755426566 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚197915709284864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211906833810432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124549975 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚178979628054528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚178979611015168 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83478529 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚189018143772160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚197915709284864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212453951 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51419823 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202851637975552 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8897565512704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9055171894784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 trainer/global_step‚4799 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8871757414706 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884211030047903e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚391 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87903976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32058706 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚187851144780518 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212479649 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚189018143772160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212479649 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83477884 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124575671 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124549975 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624œSR”î&ìèM F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87903976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32058706 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚187851144780518 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212479649 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚189018143772160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212479649 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83477884 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124575671 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124549975 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚189018125769216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87903978 _runtime‚4573.755426566 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚197915709284864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211906833810432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124549975 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚178979628054528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚178979611015168 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83478529 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚189018143772160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚197915709284864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212453951 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51419823 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202851637975552 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8897565512704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9055171894784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 trainer/global_step‚4799 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8871757414706 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884211030047903e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚391‡Ì\Ö2í- _wandbruntime‚4573  _runtime‚4573ã…Þ>[îÕ 0loss/FBt1/assignment_loss‚0.1666429558762401 1loss/FBt2/assignment_loss‚0.05715681907217008 1loss/SRqqt3/detection_loss‚1.8956752281706288 /loss/FBt4/detection_loss‚2.5039268996980457 epoch‚3 2loss/SRqqt1/assignment_loss‚1.4904508776962757 trainer/global_step‚4799 /loss/FRt1/assignment_loss‚3.657252218042101 1loss/SRqqt3/assignment_loss‚1.543264400108524 %loss/total_loss‚51.34401136922652 _runtime‚4573.756616438 0loss/FRt4/assignment_loss‚3.7544579131262643 1loss/FBt4/assignment_loss‚0.27532377776353517 /loss/FRt3/assignment_loss‚3.997482615273173 .loss/FRt2/detection_loss‚2.260451057516947 1loss/SRqqt2/detection_loss‚1.0654434615106725 2loss/SRqqt4/assignment_loss‚1.7193580999666331 1loss/FBt3/assignment_loss‚0.04650026932358742 /loss/FRt3/detection_loss‚2.7202191265617928 .loss/FRt4/detection_loss‚2.419348611150469 .loss/FBt1/detection_loss‚3.085718505912357 /loss/FBt2/detection_loss‚0.5512627538881804 2loss/SRqqt2/assignment_loss‚1.2686523059939048 /loss/FRt1/detection_loss‚2.3520850837230682 1loss/SRqqt1/detection_loss‚2.4382947385311127 % _timestamp‚1.7884211030067537e+09 0loss/FRt2/assignment_loss‚1.9809358381643527 1loss/SRqqt4/detection_loss‚2.1568726951978645 .loss/FBt3/detection_loss‚7.937235116958618 _step‚392l g}`ïÚ 2loss/SRqqt2/assignment_loss‚1.2686523059939048 /loss/FRt1/detection_loss‚2.3520850837230682 1loss/SRqqt1/detection_loss‚2.4382947385311127 % _timestamp‚1.7884211030067537e+09 0loss/FRt2/assignment_loss‚1.9809358381643527 1loss/SRqqt4/detection_loss‚2.1568726951978645 .loss/FBt3/detection_loss‚7.937235116958618 _step‚392 0loss/FBt1/assignment_loss‚0.1666429558762401 1loss/FBt2/assignment_loss‚0.05715681907217008 1loss/SRqqt3/detection_loss‚1.8956752281706288 /loss/FBt4/detection_loss‚2.5039268996980457 epoch‚3 2loss/SRqqt1/assignment_loss‚1.4904508776962757 trainer/global_step‚4799 /loss/FRt1/assignment_loss‚3.657252218042101 1loss/SRqqt3/assignment_loss‚1.543264400108524 %loss/total_loss‚51.34401136922652 _runtime‚4573.756616438 0loss/FRt4/assignment_loss‚3.7544579131262643 1loss/FBt4/assignment_loss‚0.27532377776353517 /loss/FRt3/assignment_loss‚3.997482615273173 .loss/FRt2/detection_loss‚2.260451057516947 1loss/SRqqt2/detection_loss‚1.0654434615106725 2loss/SRqqt4/assignment_loss‚1.7193580999666331 1loss/FBt3/assignment_loss‚0.04650026932358742 /loss/FRt3/detection_loss‚2.7202191265617928 .loss/FRt4/detection_loss‚2.419348611150469 .loss/FBt1/detection_loss‚3.085718505912357 /loss/FBt2/detection_loss‚0.5512627538881804ˆ6Žð‚(:ƒ ðÇäÔý•¿x gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h¸’³sñ‚(:è ðÇäÔòÍ£{ cpu‚25.242539677884942 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.in‚ 0.05078125 disk.dm-0.in‚ 4.9765625 disk.dm-1.in‚7.3125 disk.rbd0.in‚0% disk./.usageGB‚1097.4073028564453 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0* proc.memory.percent‚17.399263381958008 network.recv‚ 1.640093e+06 disk.rbd4.in‚0 disk.rbd2.in‚0# memory_percent‚98.9478349685669 disk.rbd3.in‚0 disk.nvme0n1.in‚ 4.9765625$ disk.nvme0n1.out‚20029.966796875 disk.sda2.in‚0" proc.memory.rssMB‚ 2850.6953125) disk./.usagePercent‚73.64792488075373 disk.sda1.out‚0 disk.rbd1.out‚ 2.20947265625! disk.dm-0.out‚20029.966796875 disk.dm-1.out‚ 262.90625 disk.rbd2.out‚159.7998046875 disk.sda3.in‚7.3125 disk.sda3.out‚ 262.90625( proc.memory.availableMB‚ 172.38671875 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda.out‚ 262.9453125 network.sent‚ 1.0871442e+07 disk.drbd1777.in‚ 1.03515625& disk.nvme0c0n1.out‚20029.966796875ω8^ò‚(:… ÿÇäԡΚx gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çóÍ yó‚(:î ÿÇäÔ‰²Óz cpu‚25.24192123395042$ disk./.usageGB‚1097.407283782959 disk.sda3.out‚ 263.16015625$ disk.nvme0n1.out‚20030.236328125 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd4.in‚0) disk./.usagePercent‚73.64792360071607 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625( proc.memory.availableMB‚ 172.33984375 disk.sda3.in‚7.3125 disk.dm-1.out‚ 263.16015625 disk.rbd0.in‚0 disk.rbd4.out‚0$ memory_percent‚98.94812107086182" proc.memory.rssMB‚ 2850.6953125 proc.cpu.threads‚17 network.sent‚ 1.0979782e+07 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚20030.236328125 network.recv‚ 1.646263e+06& disk.nvme0c0n1.out‚20030.236328125 disk.rbd1.out‚ 2.20947265625 disk.nvme0c0n1.in‚ 4.9765625 disk.nvme0n1.in‚ 4.9765625 disk.sda.out‚ 263.19921875 disk.dm-0.in‚ 4.9765625 disk.rbd2.out‚159.8779296875* proc.memory.percent‚17.399263381958008ÒjñS›ô‚(: ŽÈäԛȰx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OZžœ{õ‚(:ð ŽÈäÔ‰úÌz network.recv‚ 1.650147e+06 disk.dm-1.out‚ 263.8203125 disk.rbd2.in‚0 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.rbd0.out‚0* proc.memory.percent‚17.399263381958008 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 2.20947265625# disk.dm-0.out‚20031.24462890625 disk.rbd2.out‚161.3896484375 disk.rbd4.out‚0( disk.nvme0c0n1.out‚20031.24462890625& disk.nvme0n1.out‚20031.24462890625( proc.memory.availableMB‚ 172.33984375 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9765625) disk./.usagePercent‚73.64792360071607$ disk./.usageGB‚1097.407283782959 disk.rbd3.in‚0 disk.sda3.out‚ 263.8203125" proc.memory.rssMB‚ 2850.6953125 network.sent‚ 1.0985645e+07 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 4.9765625 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 cpu‚25.24130019207444 disk.sda.out‚ 263.859375$ memory_percent‚98.94812107086182 proc.cpu.threads‚17À½¨?ö‚(:„ ÈäÔƒ– x gpu.0.gpu‚47.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0||¾÷‚(:õ ÈäÔ™±Á{ disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.sda1.in‚ 0.050781M„Õ7ÿ25 disk.dm-0.in‚ 4.9765625 disk.sda2.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚20039.77978515625 disk.dm-1.in‚7.3125) proc.memory.percent‚17.45448112487793 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9765625& disk.nvme0n1.out‚20039.77978515625 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 264.21484375 disk.sda2.out‚ 0.0390625 cpu‚25.254876790434807 network.recv‚ 1.655247e+06) disk./.usagePercent‚73.64846377661331 disk.rbd4.out‚0 disk.dm-1.out‚ 264.21484375$ memory_percent‚99.00457859039307( proc.memory.availableMB‚ 163.08984375% disk./.usageGB‚1097.4153327941895 disk.nvme0c0n1.in‚ 4.9765625( disk.nvme0c0n1.out‚20039.77978515625 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 2.20947265625" proc.memory.rssMB‚ 2859.7421875 network.sent‚ 1.0991945e+07 disk.rbd2.out‚161.4677734375 disk.sda.out‚ 264.25390625 disk.sda3.in‚7.3125 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda1.out‚0üéŸ:ø‚(:„ ¬ÈäÔx gpu.0.gpu‚10.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þöY~ù‚(:ó ¬ÈäÔÛ¨| network.sent‚ 1.0997241e+07 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 264.76953125 disk.rbd3.out‚0& proc.memory.availableMB‚ 163.296875) disk./.usagePercent‚73.64846454463591% disk./.usageGB‚1097.4153442382812# disk.dm-0.out‚20040.63916015625 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda3.out‚ 264.76953125& disk.nvme0n1.out‚20040.63916015625 disk.rbd0.in‚0 network.recv‚ 1.658889e+06 disk.dm-0.in‚ 4.9765625 disk.sda2.in‚0" proc.memory.rssMB‚ 2859.7421875 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.in‚7.3125$ memory_percent‚99.00331497192383 cpu‚25.268798263146525 disk.rbd2.out‚161.4677734375 disk.nvme0c0n1.in‚ 4.9765625( disk.nvme0c0n1.out‚20040.63916015625 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 264.80859375 disk.rbd1.in‚0 disk.rbd1.out‚ 2.20947265625) proc.memory.percent‚17.45448112487793 disk.rbd0.out‚0 disk.nvme0n1.in‚ 4.9765625 disk.dm-1.in‚7.3125 disk.sda.in‚ 7.36328125 disk.drbd1777.out‚0ÝÇØú‚(:„ »ÈäÔÂ¥ºx gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ÑÑð|û‚(:ñ »ÈäÔñîŽ{ disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚ 4.9765625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.9765625$ memory_percent‚99.00226593017578 proc.cpu.threads‚17$ disk./.usageGB‚1097.540916442871 disk.rbd3.out‚0 disk.sda2.out‚ 0.0390625! disk.rbd2.out‚161.57763671875" proc.memory.rssMB‚ 2859.7421875 cpu‚25.28286243316459 network.recv‚ 1.663601e+06 disk.sda3.in‚7.3125 disk.rbd1.out‚ 2.20947265625 disk.rbd4.out‚0 disk.rbd2.in‚0% proc.memory.availableMB‚ 163.46875 disk.rbd3.in‚0 disk.sda.out‚ 265.29296875 disk.sda3.out‚ 265.25390625 disk.dm-0.in‚ 4.9765625 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 network.sent‚ 1.1003413e+07 disk.rbd1.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚20040.96923828125) proc.memory.percent‚17.45448112487793 disk.sda.in‚ 7.36328125 disk.dm-1.out‚ 265.25390625 disk.drbd1777.out‚0) disk./.usagePercent‚73.65689180064804& disk.nvme0n1.out‚20040.96923828125# disk.dm-0.out‚20040.96923828125 disk.rbd4.in‚0dз›2ü- _wandbruntime‚4659  _runtime‚4659Q=ãµF*ýÀT XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 6893338624 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 549453824 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 9141780265984 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚181499394011136 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚262 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚89273726 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚366 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 184860760 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 167821400 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 537294673 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚32555418 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 6893338624 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 189713920 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚191679208456704 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31397 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8956914817614 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8956746996214 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚200662200309248 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 171710976 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6343884800 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 215175807 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚89273724 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚191679190453760 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20203 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 125902081 _runtime‚4659.009545102 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 189713920 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚200662010595328 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚191679190453760 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8982991852544 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 215175807 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚561 _step‚393 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 125902081 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 149153280 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 6893338624 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚299 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 125881880 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1261 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20201 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚200662200309248 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚84786891 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1243 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 541892608 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚89273726 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4835 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚299 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚299 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚191679208456704Ž­5³K*þÅT ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20203 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚84787257 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚190456325868110 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚181499411050496 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 549453824 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 215155604 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚214818712755200 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚23 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚364 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚205677105582592 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚190456141007350 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚299 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚191679208456704 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 6893338624 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 549453824 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 9141780265984 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚181499394011136 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚262 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚89273726 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚366 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 184860760 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 167821400 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 537294673 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚32555418 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 6893338624 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 189713920 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚191679208456704 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31397 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8956914817614 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8956746996214 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚200662200309248 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 171710976 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6343884800 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 215175807 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚89273724 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚191679190453760 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20203 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 125902081 _runtime‚4659.009545102 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 189713920 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚200662010595328 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚191679190453760 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8982991852544 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 215175807 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚561 _step‚393 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 125902081 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 149153280 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 6893338624 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚299 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 125881880 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1261 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20201 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚200662200309248 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚84786891 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1243 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 541892608 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚89273726 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4835 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚299 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31397 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚190481285133494 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 8957305994510 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 215199019 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8983212936704 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 215199019 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚52240801 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31397 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337968128 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚205704496634368 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚191705331243520 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚89288660 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚23 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 549453824 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6739344896 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20203 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31168 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20201 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20201 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8983212936704 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8957138173110 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 6893338624 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 549453824 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚200688733894144 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚20‹ºA¡0688733894144 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31168 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚84801141 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 171710976 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 9142173061120 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚181524163999744 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚181524146960384 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚561 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 189713920 _runtime‚4660.042504091 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9142023907840 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚561 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚561 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 215178816 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚200688544180224 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚214846693635584 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6343884800 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚299 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 173093376 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6739344896 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚200688544180224 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚191705349246464 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚89288662 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3428 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚365 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 6893338624 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚52240803 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31397 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚190481285133494 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 8957305994510 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 215199019 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8983212936704 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 215199019 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚52240801 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31397 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337968128 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚205704496634368 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚191705331243520 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚89288660 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚23 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 549453824 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6739344896 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20203 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31168 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20201 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20201 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8983212936704 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8957138173110 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 6893338624 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 549453824 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚200688733894144 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚200688733894144 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31168 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚84801141 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 171710976 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 9142173061120 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚181524163999744 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚181524146960384 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚561 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 189713920 _runtime‚4660.042504091 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9142023907840 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚561 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚561 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 215178816 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚200688544180224 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚214846693635584 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6343884800 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚299 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 173093376 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6739344896 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚200688544180224 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚191705349246464 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚89288662 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3428 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚365 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 6893338624 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚52240803 B&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4475317597389221 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2495892345905304 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.26209181547164917 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.42030948400497437 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3666880428791046 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.29598215222358704 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01075268816202879 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3900177478790283 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4423757493495941 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.42580628395080566 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.41454339027404785 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3295454680919647 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4057539403438568 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.860040545463562 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1511618196964264 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_7‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10113272070884705 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06723745912313461 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13248765468597412 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08683644980192184 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.38450708985328674 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.44760629534721375 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 6 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.225862056016922 *jet/accuracy_1_of_3‚0.8142759203910828 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2708333432674408 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 8 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.02083333395421505 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4308030307292938 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9892322421073914 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3567529022693634 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.10000000149011612 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.07183908671140671 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3830409348011017 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01031745970249176 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.36427202820777893 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.09887994080781937 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_4‚1 particle/accuracy_3_of_5‚1 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9861299395561218 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1790882647037506 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚46.82740273388597 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4291694760322571 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4343486726284027 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6338579058647156 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9878193140029907 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.43743619322776794 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30116063356399536 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9877192974090576 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9845048189163208 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23616492748260498 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 jet/accuracy_5_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.000933706818614155 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.016139285638928413 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,jet/accuracy_3_of_3‚0.021076081320643425 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4509446322917938 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 *jet/accuracy_1_of_6‚0.9615384340286255 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 _step‚395 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21947434544563293 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4479433298110962 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33752796053886414 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206268787384 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.45677170157432556 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.38897010684013367 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.42307692766189575 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1219419315457344 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.169288769364357 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996392130851746 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12568789720535278 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9931973218917847 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9919263124465942 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4438765048980713 )jet/accuracy_1_of_5‚0.937711238861084 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066193699836731 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9859419465065002 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_6‚0.057692307978868484 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9910506010055542 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9898577928543091 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.109089195728302 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4444444477558136 particle/accuracy_2_of_5‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3263889253139496 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.44607794284820557 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3527778089046478 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 _runtime‚4673.788739948 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4120370149612427 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.23180556297302246 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.027070919051766396 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.39387714862823486 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3410852551460266 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.19223853945732117 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 jet/accuracy_4_of_6‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9899574518203735 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 particle/accuracy_2_of_7‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.27684158086776733 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03563748300075531 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01429767906665802 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ( Purity/1FRt3SRqqt1FBt/FRt_purity‚0.5 % _timestamp‚1.7884212030366182e+09 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43062442541122437 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07587739080190659 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961341619491577 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09564121812582016 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09844261407852173 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10493439435958862 *jet/accuracy_1_of_1‚0.4426150321960449 +particle/specificity‚0.9545875191688538 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16651666164398193 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4191918969154358 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 nÍbý R=&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 particle/accuracy_4_of_4‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1980200707912445 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9780274629592896 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0007936508045531809 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005881804041564465 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878787517547607 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43633008003234863 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4218754470348358 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10470010340213776 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.42616525292396545 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3866666853427887 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24421104788780212 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.05967078357934952 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41634541749954224 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4297619163990021 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3611111044883728 particle/accuracy_1_of_6‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 epoch‚3 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9876548051834106 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1088157445192337 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.42053288221359253 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4296630322933197 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.40248173475265503 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.06228070333600044 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.26977482438087463 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.231634259223938 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02600426785647869 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14607852697372437 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05872460454702377 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1944444328546524 ,"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.45779576897621155 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,jet/accuracy_3_of_4‚0.041715383529663086 particle/accuracy_2_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.44667884707450867 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9790578484535217 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 'particle/f_score‚0.3835369646549225 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 jet/accuracy_4_of_7‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4215736985206604 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.143333300948143 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.397916704416275 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 -jet/accuracy_4_of_4‚0.0027557318098843098 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.49122804403305054 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11756997555494308 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_2_of_2‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03184523805975914 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.20277778804302216 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4326271116733551 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15736153721809387 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0028964863158762455 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.08943097293376923 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19461485743522644 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4140528440475464 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.06704980880022049 *jet/accuracy_2_of_4‚0.2996107041835785 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4069124162197113 (particle/accuracy‚0.8516865372657776 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_5_of_6‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08921398222446442 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1109195351600647 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_5_of_5‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10266797989606857 particle/accuracy_2_of_3‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09906569123268127 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08921398222446442 particle/accuracy_1_of_3‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.44304901361465454 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.13425926864147186 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.38524335622787476 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9952381253242493 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14595670998096466 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9953096508979797 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2611423432826996 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.39691591262817383 particle/accuracy_7_of_7‚0.5 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.21756331622600555 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,particle/sensitivity‚0.29564595222473145 jet/accuracy_7_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11376672983169556 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1078282818198204 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11559320986270905 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9899574518203735 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9915457367897034 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3496527671813965 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3623192012310028 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.41372549533843994 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.24074077606201172 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011659108102321625 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06651155650615692 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02425875887274742 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.06444444507360458 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_6‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4267262816429138 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14966602623462677 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.009803921915590763 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3035714328289032 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15011228621006012 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982138872146606 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1629558652639389 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 jet/accuracy_2_of_7‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4492770731449127 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.25833335518836975 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4316268861293793 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.059080012142658234 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9684028029441833 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.44232070446014404 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.6972731351852417 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10363686084747314 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.07945311814546585 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9877777099609375 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13235001266002655 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.12882882356643677 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.42682698369026184 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.41339871287345886 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09929077327251434 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6622340679168701 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4104336202144623 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2563987970352173 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 /"Purity/1FRt1SRqqt2FBt/event_purity‚0.015625 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.35197368264198303 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.042002540081739426 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 trainer/global_step‚4835 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8082578778266907 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.024338088929653168 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.44097694754600525 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.016156461089849472 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3903903365135193 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4866666793823242 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.349074125289917 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.010440584272146225 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15173739194869995 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971110820770264 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.08071279525756836 particle/accuracy_1_of_1‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.24867048859596252 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.37051284313201904 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878054857254028 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4115433096885681 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4192034900188446 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4583333432674408 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.43944990634918213 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 jet/accuracy_1_of_7‚0.5ìm|å-†­§ ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 trainer/global_step‚4835 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8082578778266907 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.024338088929653168 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.44097694754600525 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.016156461089849472 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3903903365135193 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4866666793823242 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.349074125289917 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.010440584272146225 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15173739194869995 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971110820770264 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.08071279525756836 particle/accuracy_1_of_1‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.24867048859596252 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.37051284313201904 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878054857254028 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4115433096885681 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4192034900188446 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4583333432674408 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.43944990634918213 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 jet/accuracy_1_of_7‚0.5 8 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.02083333395421505 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9888569712638855 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4509446322917938 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09432084113359451 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121211886405945 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9733333587646484 jet/accuracy_3_of_7‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.40223804116249084 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4043216407299042 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12062718719244003 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.477011501789093 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9845545291900635 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4475317597389221 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2495892345905304 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.26209181547164917 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.42030948400497437 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3666880428791046 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.29598215222358704 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01075268816202879 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3900177478790283 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4423757493495941 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.42580628395080566 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.41454339027404785 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3295454680919647 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4057539403438568 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.860040545463562 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1511618196964264 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_7‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10113272070884705 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06723745912313461 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13248765468597412 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08683644980192184 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.38450708985328674 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.44760629534721375 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 6 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.225862056016922 *jet/accuracy_1_of_3‚0.8142759203910828 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2708333432674408 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 8 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.02083333395421505 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4308030307292938 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9892322421073914 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3567529022693634 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.10000000149011612 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.07183908671140671 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3830409348011017 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01031745970249176 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.36427202820777893 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.09887994080781937 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_4‚1 particle/accuracy_3_of_5‚1 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9861299395561218 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1790882647037506 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚46.82740273388597 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4291694760322571 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4343486726284027 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6338579058647156 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9878193140029907 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.43743619322776794 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30116063356399536 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9877192974090576 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9845048189163208 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23616492748260498 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 jet/accuracy_5_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 & Purity/0FRt0SRqqt1FBt/FBt_p  mÏeurity‚1 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.000933706818614155 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.016139285638928413 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,jet/accuracy_3_of_3‚0.021076081320643425 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4509446322917938 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 *jet/accuracy_1_of_6‚0.9615384340286255 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 _step‚395 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21947434544563293 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4479433298110962 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33752796053886414 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206268787384 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.45677170157432556 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.38897010684013367 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.42307692766189575 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1219419315457344 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.169288769364357 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996392130851746 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12568789720535278 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9931973218917847 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9919263124465942 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4438765048980713 )jet/accuracy_1_of_5‚0.937711238861084 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066193699836731 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9859419465065002 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_6‚0.057692307978868484 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9910506010055542 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9898577928543091 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.109089195728302 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4444444477558136 particle/accuracy_2_of_5‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3263889253139496 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.44607794284820557 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3527778089046478 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 _runtime‚4673.788739948 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4120370149612427 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.23180556297302246 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.027070919051766396 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.39387714862823486 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3410852551460266 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.19223853945732117 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 jet/accuracy_4_of_6‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9899574518203735 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 particle/accuracy_2_of_7‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.27684158086776733 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03563748300075531 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01429767906665802 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ( Purity/1FRt3SRqqt1FBt/FRt_purity‚0.5 % _timestamp‚1.7884212030366182e+09 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43062442541122437 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07587739080190659 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961341619491577 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09564121812582016 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09844261407852173 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10493439435958862 *jet/accuracy_1_of_1‚0.4426150321960449 +particle/specificity‚0.9545875191688538 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16651666164398193 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4191918969154358 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 particle/accuracy_4_of_4‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1980200707912445 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9780274629592896 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0007936508045531809 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005881804041564465 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878787517547607 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43633008003234863 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4218754470348358 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10470010340213776 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.42616525292396545 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3866666853427887 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24421104788780212 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.05967078357934952 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41634541749954224 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4297619163990021 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3611111044883728 particle/accuracy_1_of_6‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 epoch‚3 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9876548051834106 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1088157445192337 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.42053288221359253 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4296630322933197 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.40248173475265503 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.06228070333600044 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.26977482438087463 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.231634259223938 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02600426785647869 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14607852697372437 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05872460454702377 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1944444328546524 ,"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.45779576897621155 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,jet/accuracy_3_of_4‚0.041715383529663086 particle/accuracy_2_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.44667884707450867 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9790578484535217 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 'particle/f_score‚0.3835369646549225 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 jet/accuracy_4_of_7‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4215736985206604 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.143333300948143 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.397916704416275 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 -jet/accuracy_4_of_4‚0.0027557318098843098 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.49122804403305054 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11756997555494308 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_2_of_2‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03184523805975914 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.20277778804302216 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4326271116733551 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15736153721809387 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0028964863158762455 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.08943097293376923 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19461485743522644 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4140528440475464 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.06704980880022049 *jet/accuracy_2_of_4‚0.2996107041835785 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4069124162197113 (particle/accuracy‚0.8516865372657776 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_5_of_6‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08921398222446442 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1109195351600647 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_5_of_5‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10266797989606857 particle/accuracy_2_of_3‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09906569123268127 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08921398222446442 particle/accuracy_1_of_3‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.44304901361465454 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.13425926864147186 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.38524335622787476 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9952381253242493 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14595670998096466 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9953096508979797 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2611423432826996 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.39691591262817383 particle/accuracy_7_of_7‚0.5 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.21756331622600555 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,particle/sensitivity‚0.29564595222473145 jet/accuracy_7_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11376672983169556 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1078282818198204 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11559320986270905 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9899574518203735 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9915457367897034 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3496527671813965 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3623192012310028 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.41372549533843994 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.24074077606201172 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011659108102321625 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06651155650615692 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02425875887274742 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.06444444507360458 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_6‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4267262816429138 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14966602623462677 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.009803921915590763 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3035714328289032 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15011228621006012 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982138872146606 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1629558652639389 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 jet/accuracy_2_of_7‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4492770731449127 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.25833335518836975 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4316268861293793 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.059080012142658234 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9684028029441833 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.44232070446014404 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.6972731351852417 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10363686084747314 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.07945311814546585 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9877777099609375 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13235001266002655 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.12882882356643677 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.42682698369026184 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.41339871287345886 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09929077327251434 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6622340679168701 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4104336202144623 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2563987970352173 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 /"Purity/1FRt1SRqqt2FBt/event_purity‚0.015625 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.35197368264198303 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.042002540081739426 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25‹]&$Ó‡‚(:‚ ÙÈäÔÂðžx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.646 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ryP€ˆ‚(:õ ÙÈäÔ©žÍz disk.rbd4.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚20406.28759765625( proc.memory.availableMB‚ 504.28515625" proc.memory.rssMB‚ 2862.6953125* proc.memory.percent‚17.472505569458008 disk.nvme0c0n1.in‚ 4.9765625 disk.sda3.in‚7.3125 disk.sda2.in‚0 disk.dm-1.in‚7.3125! disk.rbd2.out‚163.22607421875 disk.drbd1777.in‚ 1.03515625 disk.rbd0.in‚0 disk.rbd3.in‚0) disk./.usagePercent‚73.66530881635879% disk./.usageGB‚1097.6663360595703 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚20406.28759765625 disk.sda.in‚ 7.36328125 disk.sda.out‚ 266.30078125$ memory_percent‚96.92208766937256 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 266.26171875 disk.dm-1.out‚ 266.26171875 disk.rbd0.out‚0 cpu‚25.3043310026063 disk.rbd4.in‚0& disk.nvme0n1.out‚20406.28759765625 disk.sda2.out‚ 0.0390625 network.recv‚ 1.674591e+06 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 2.20947265625 disk.nvme0n1.in‚ 4.9765625 disk.dm-0.in‚ 4.9765625 disk.rbd2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.1117715e+07rM5°2‰- _wandbruntime‚4688  _runtime‚4688WÈpßÅŠ¿ trainer/global_step‚4849 % _timestamp‚1.7884212182278714e+09 _runtime‚4688.977793718 _step‚396 % lr-AdamW/pg1‚0.000600433610414427 % lr-AdamW/pg2‚0.000600433610414427ëýµÊ‹Ä % lr-AdamW/pg1‚0.000600433610414427 % lr-AdamW/pg2‚0.000600433610414427 trainer/global_step‚4849 % _timestamp‚1.7884212182278714e+09 _runtime‚4688.977793718 _step‚396ŒîÔžo2Œ- _wandbruntime‚4688  _runtime‚4688FÅU«ùÙO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126354321 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32713189 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _step‚397 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216067498 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201555716605952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89713175 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52491948 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126328651 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚192543554783232 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126328651 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216041826 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201555962812416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9012161822720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201555716605952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚192543554783232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9012161822720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191304104130028 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 % _timestamp‚1.7884212182286494e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚192543572786176 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126354321 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4688.978911032 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32712546 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.currenÔöyæt‚ 549453824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9012390026240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201555962812416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85204494 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚206596097580032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚192543572786176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9171445280768 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚182317884292096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191303863441520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9012390026240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚206596121520128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9171716199424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89713177 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215767542860800 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚182317867252736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8985996188784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32712546 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9012390026240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201555962812416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85204494 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚206596097580032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚192543572786176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9171445280768 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚182317884292096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191303863441520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9012390026240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚206596121520128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9171716199424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89713177 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215767542860800 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚182317867252736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8985996188784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201555716605952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89713175 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52491948 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126328651 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚192543554783232 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126328651 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216041826 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201555962812416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9012161822720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201555716605952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚192543554783232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9012161822720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191304104130028 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 % _timestamp‚1.7884212182286494e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚192543572786176 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126354321 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4688.978911032J¨g72- _wandbruntime‚4689  _runtime‚4689ºê„é&ãM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9013999517184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚206637267276288 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126354202 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216085096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89730894 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚182354169710592 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216110794 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9014227734016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192581910730240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52502303 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85220956 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89730896 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32718653 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4849 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9013999517184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9173329819136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8988052237740 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89730894 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85221592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32719287 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215810894038016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 % _timestamp‚1.7884212190908785e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201595892244480 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚192581910730240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216085096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126379898 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52502305 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚182354186749952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191342238987692 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9173602821632 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9014227734016 _step‚398 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191341998298976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚192581892727296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216110794 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215810597095424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚206637291216384 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚182354186749952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191342238987692 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9173602821632 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9014227734016 _step‚398 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191341998298976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚192581892727296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216110794 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215810597095424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚206637291216384 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192581910730240 REDeviceStatsMonitor.on_train_batch_end/inactive_s¸âÐEö plit.large_pool.freed‚52502303 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85220956 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89730896 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32718653 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4849 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9013999517184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9173329819136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8988052237740 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89730894 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85221592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32719287 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215810894038016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 % _timestamp‚1.7884212190908785e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201595892244480 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624Žµ€”Ñ2’- _wandbruntime‚4689  _runtime‚4689{ƒuX“Ò /loss/FBt2/detection_loss‚0.5761748531931326 _runtime‚4689.842638763 /loss/FRt3/assignment_loss‚3.551660441770786 0loss/FBt3/assignment_loss‚0.2714732133317739 trainer/global_step‚4849 .loss/FBt4/detection_loss‚2.852782040834427 $ _timestamp‚1.788421219092765e+09 1loss/SRqqt1/detection_loss‚2.3515222208840507 .loss/FBt1/detection_loss‚2.290011131763458 epoch‚4 _step‚399 2loss/SRqqt1/assignment_loss‚2.0940275792564664 2loss/SRqqt4/assignment_loss‚1.6303264123315995 1loss/FBt2/assignment_loss‚0.15253446076673632 %loss/total_loss‚48.11592454103544 0loss/FBt1/assignment_loss‚0.5220789869781584 0loss/FBt4/assignment_loss‚0.1137178052613308 1loss/SRqqt2/detection_loss‚1.5666450155170069 0loss/FRt4/assignment_loss‚3.7761637197958455 2loss/SRqqt3/assignment_loss‚1.7841989876794033 /loss/FRt4/detection_loss‚2.3715948647743965 /loss/FBt3/detection_loss‚3.0508203878998756 /loss/FRt2/assignment_loss‚2.176764305900125 2loss/SRqqt2/assignment_loss‚1.3958010875174534 1loss/SRqqt3/detection_loss‚1.5907023803132478 /loss/FRt1/assignment_loss‚4.357165727615357 /loss/FRt1/detection_loss‚2.3932017283141613 .loss/FRt2/detection_loss‚2.466855998753625 /loss/FRt3/detection_loss‚2.5990583358741386 0loss/SRqqt4/detection_loss‚2.180642854708892ÈÕ%]”× 0loss/SRqqt4/detection_loss‚2.180642854708892 /loss/FBt2/detection_loss‚0.5761748531931326 _runtime‚4689.842638763 /loss/FRt3/assignment_loss‚3.551660441770786 0loss/FBt3/assignment_loss‚0.2714732133317739 trainer/global_step‚4849 .loss/FBt4/detection_loss‚2.852782040834427 $ _timestamp‚1.788421219092765e+09 1loss/SRqqt1/detection_loss‚2.3515222208840507 .loss/FBt1/detection_loss‚2.290011131763458 epoch‚4 _step‚399 2loss/SRqqt1/assignment_loss‚2.0940275792564664 2loss/SRqqt4/assignment_loss‚1.6303264123315995 1loss/FBt2/assignment_loss‚0.15253446076673632 %loss/total_loss‚48.11592454103544 0loss/FBt1/assignment_loss‚0.5220789869781584 0loss/FBt4/assignment_loss‚0.1137178052613308 1loss/SRqqt2/detection_loss‚1.5666450155170069 0loss/FRt4/assignment_loss‚3.7761637197958455 2loss/SRqqt3/assignment_loss‚1.7841989876794033 /loss/FRt4/detection_loss‚2.3715948647743965 /loss/FBt3/detection_loss‚3.0508203878998756 /loss/FRt2/assignment_loss‚2.176764305900125 2loss/SRqqt2/assignment_loss‚1.3958010875174534 1loss/SRqqt3/detection_loss‚1.5907023803132478 /loss/FRt1/assignment_loss‚4.357165727615357 /loss/FRt1/detection_loss‚2.3932017283141613 .loss/FRt2/detection_loss‚2.466855998753625 /loss/FRt3/detection_loss‚2.5990583358741386ìËñ÷•‚(:… èÈäÔõôŽx gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.784' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1568 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$yzÒ~–‚(:ó èÈäÔ¢à¼z( proc.memory.availableMB‚ 504.49609375 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 266.91015625 disk.rbd1.out‚ 2.20947265625 disk.nvme0n1.in‚ 4.9765625 disk.dm-0.in‚ 4.9765625 disk.dm-1.out‚ 266.87109375 disk.rbd0.in‚0 network.recv‚ 1.6795e+06% disk./.usageGB‚1097.9157752990723 disk.dm-1.in‚7.3125 disk.rbd0.out‚0$ memory_percent‚96.92080020904541* proc.memory.percent‚17.472505569458008 cpu‚25.30347148810113& disk.nvme0n1.out‚20514.73193359375# disk.dm-0.out‚20514.73193359375 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 1.124102e+07 disk.rbd1.in‚0 disk.sda2.in‚0! disk.rbd2.out‚163.22607421875( disk.nvme0c0n1.out‚20514.73193359375) disk./.usagePercent‚73.68204889301524 disk.sda3.in‚7.3125 disk.rbd4.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2862.6953125 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 266.87109375 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625ÿìHàš—‚(: ÷ÈäÔª‚™x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.648' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.729599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=lÂx˜‚(:í ÷ÈäÔÓ§åz disk.sda3.in‚7.3125 disk.dm-0.in‚ 4.9765625 disk.nvme0c0n1.in‚ 4.9765625# disk.dm-0.out‚20673.57568359375 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2862.7578125% disk./.usageGB‚1097.9157752990723 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 267.359375 disk.dm-1.in‚7.3125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚20673.57568359375 cpu‚25.302718043491357 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.dm-1.out‚ 267.359375 disk.rbd2.out‚ 163.33203125 network.sent‚ 1.1349434e+07 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 2.20947265625 disk.rbd4.in‚0 disk.rbd4.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.68204889301524 disk.rbd3.in‚0' proc.memory.availableMB‚ 504.3515625) proc.memory.percent‚17.47288703918457 network.recv‚ 1.685733e+06 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0$ memory_percent‚96.92168235778809 disk.sda.out‚ 267.3984375 disk.nvme0n1.in‚ 4.9765625& disk.nvme0n1.out‚20673.57568359375íà·ë™‚(:… †ÉäÔÛ´¢x gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û ê·zš‚(:ï †ÉäÔÓÄž|( disk.nvme0c0n1.out‚20773.22216796875 network.recv‚ 1.68984e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 2.20947265625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚96.92463874816895 disk.nvme0n1.in‚ 4.9765625" proc.memory.rssMB‚ 2862.7578125 network.sent‚ 1.1355373e+07 disk.sda2.in‚0 disk.rbd2.out‚ 163.33203125 disk.drbd1777.out‚0 disk.rbd4.in‚0' proc.memory.availableMB‚ 503.8671875 cpu‚25.3019042327077& disk.nvme0n1.out‚20773.22216796875 disk.sda2.out‚ 0.0390625 disk.rbd4.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 267.95703125 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625 disk.sda3.in‚7.3125# disk.dm-0.out‚20773.22216796875 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 267.91796875 disk.sda3.out‚ 267.91796875) proc.memory.percent‚17.47288703918457) disk./.usagePercent‚73.68204889301524 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.in‚ 0.05078125% disk./.usageGB‚1097.9157752990723 disk.dm-0.in‚ 4.9765625 disk.rbd0.out‚0ä9œ2›- _wandbruntime‚4734  _runtime‚4734‰š/PÇœÁ & lr-AdamW/pg1‚0.0005953868683300341 & lr-AdamW/pg2‚0.0005953868683300341 trainer/global_step‚4899 % _timestamp‚1.7884212636283998e+09 _runtime‚4734.378304149 _step‚400죌‚ÌÆ _step‚400 & lr-AdamW/pg1‚0.0005953868683300341 & lr-AdamW/pg2‚0.0005953868683300341 trainer/global_step‚4899 % _timestamp‚1.7884212636283998e+09 _runtime‚4734.378304149}’C2ž- _wandbruntime‚4734  _runtime‚4734æýÀ”ß'ŸÙO QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _step‚401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194460469989376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127609553 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9265062909952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127635223 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127609553 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9104289163264 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203564512946176 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚4899 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚194460451986432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9265325440000 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90599125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90599125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33031853 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9104060959744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚203564512946176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208654606332928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127635223 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90599127 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53009700 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184133007184896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184132990145536 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9077629759992 _runtime‚4734.379297249 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86042217 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193210860594036 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90599127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33032517 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9077853409140 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203564759152640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217919931772928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9104289163264 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53009698j.Ùãä' ÞO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90599127 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53009700 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184133007184896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184132990145536 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9077629759992 _runtime‚4734.379297249 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86042217 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193210860594036 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90599127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33032517 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9077853409140 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203564759152640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217919931772928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9104289163264 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53009698 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _step‚401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194460469989376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127609553 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9265062909952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127635223 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127609553 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9104289163264 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203564512946176 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚4899 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚194460451986432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9265325440000 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90599125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90599125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33031853 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9104060959744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚203564512946176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208654606332928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127635223 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666‘n¸Î˜2¡- _wandbruntime‚4735  _runtime‚4735ò­ô¢ãM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90616846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9105898654208 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9106126871040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218251948 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90616844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚184169309642752 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _runtime‚4735.236112621 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚193248995451700 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86058012 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218277646 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33038626 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚193248754762984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚203604688584704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90616846 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90616844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9105898654208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127635104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9079462159592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9079685808948 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53020053 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217962976433152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚194498807933440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53020055 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194498789930496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33037959 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218251948 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203604934804480 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194498807933440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217962681587712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9266929498624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208695776029184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194498789930496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86058681 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚4899 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127660800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208695752089088 _step‚402°C_„î&£èM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9106126871040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9105898654208 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9106126871040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218251948 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90616844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚184169309642752 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _runtime‚4735.236112621 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚193248995451700 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86058012 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218277646 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33038626 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚193248754762984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚203604688584704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90616846 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90616844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9105898654208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127635104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9079462159592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9079685808948 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53020053 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217962976433152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚194498807933440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53020055 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194498789930496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33037959 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218251948 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203604934804480 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194498807933440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217962681587712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9266929498624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208695776029184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194498789930496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86058681 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚4899 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127660800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208695752089088 _step‚402 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90616846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220375530 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33356953 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220401202 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53527450 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 % _timestamp‚1.7884213088536544e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9195960096768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚210713067205632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9196188300288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33356288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91485075 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196377367192576 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205573309286400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9358849865728 Z£Ì«ŒHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9169263331200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9195960096768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9358574752768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185948130077696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86883736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196377367192576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9169486980348 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196377349189632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91485077 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9196188300288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128916125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128890455 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195117617058044D‰çðä'²ÞO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9195960096768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9358574752768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185948130077696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86883736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196377367192576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9169486980348 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196377349189632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91485077 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9196188300288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128916125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128890455 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195117617058044 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196377349189632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚205573555492864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220375530 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33356953 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220401202 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53527450 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 % _timestamp‚1.7884213088536544e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9195960096768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚210713067205632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9196188300288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33356288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91485075 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196377367192576 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205573309286400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9358849865728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9169263331200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800•ºJH#2³- _wandbruntime‚4780  _runtime‚4780˜Òñóé&´ãM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚210754260841984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128916006 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91502794 trainer/global_step‚4949 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220418800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185984415496192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185984432535552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚205613731144704 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128941702 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚210754236901888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91502796 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚4780.463691844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205613484924928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9171319380156 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚205613731144704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9197797791232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9198026008064 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220444498 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33363025 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9360730615296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚196415705136640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86900828 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚195155511226992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220418800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚220114696611840 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86901468 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220444498 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚220114991457280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196415705136640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9197797791232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91502796 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53537805 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 _step‚406 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚196415687133696 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884213097130418e+09¨6¥R µèM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91502796 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53537805 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 _step‚406 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚196415687133696 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884213097130418e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚210754260841984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128916006 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91502794 trainer/global_step‚4949 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220418800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185984415496192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185984432535552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚205613731144704 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128941702 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚210754236901888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91502796 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚4780.463691844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205613484924928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9171319380156 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚205613731144704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9197797791232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9198026008064 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220444498 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33363025 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9360730615296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚196415705136640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86900828 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚195155511226992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 O@DeviceStatsMonitor.on_train_batch_end/re¨Ò0¸œserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220418800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚220114696611840 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86901468 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220444498 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚220114991457280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196415705136640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9197797791232–&Óãü2¶- _wandbruntime‚4780  _runtime‚47802=ÌX·Ò 0loss/FRt1/assignment_loss‚3.0604221240068092 0loss/FRt3/assignment_loss‚3.6521758333496424 .loss/FRt2/detection_loss‚1.564337263405323 /loss/FRt4/detection_loss‚2.6445617467164992 epoch‚4 trainer/global_step‚4949 0loss/FBt1/assignment_loss‚0.1136624582910112 /loss/FRt1/detection_loss‚2.4816397375021224 1loss/SRqqt2/detection_loss‚0.8622701349904982 /loss/FBt2/detection_loss‚0.7827727430930702 %loss/total_loss‚45.47764841114193 0loss/FRt2/assignment_loss‚1.8039300322532654 0loss/FBt3/assignment_loss‚0.3089938682775634 1loss/FBt4/assignment_loss‚0.13925816919427234 % _timestamp‚1.7884213097150338e+09 2loss/SRqqt3/assignment_loss‚1.9885422552802734 2loss/SRqqt4/assignment_loss‚1.9263472440450087 .loss/FBt3/detection_loss‚2.632038871447245 2loss/SRqqt1/assignment_loss‚1.6441636482874553 1loss/SRqqt3/detection_loss‚1.9722343193633216 .loss/FBt4/detection_loss‚3.502335524559021 _runtime‚4780.464880213 2loss/SRqqt2/assignment_loss‚1.0104571435308494 -loss/SRqqt4/detection_loss‚2.102001376774 _step‚407 1loss/FBt2/assignment_loss‚0.16264832124506307 1loss/SRqqt1/detection_loss‚2.2787333759996624 0loss/FRt4/assignment_loss‚3.3497420102357864 .loss/FRt3/detection_loss‚2.344240562423416 .loss/FBt1/detection_loss‚3.150139646870749GxiÁ]¸× 2loss/SRqqt2/assignment_loss‚1.0104571435308494 -loss/SRqqt4/detection_loss‚2.102001376774 _step‚407 1loss/FBt2/assignment_loss‚0.16264832124506307 1loss/SRqqt1/detection_loss‚2.2787333759996624 0loss/FRt4/assignment_loss‚3.3497420102357864 .loss/FRt3/detection_loss‚2.344240562423416 .loss/FBt1/detection_loss‚3.150139646870749 0loss/FRt1/assignment_loss‚3.0604221240068092 0loss/FRt3/assignment_loss‚3.6521758333496424 .loss/FRt2/detection_loss‚1.564337263405323 /loss/FRt4/detection_loss‚2.6445617467164992 epoch‚4 trainer/global_step‚4949 0loss/FBt1/assignment_loss‚0.1136624582910112 /loss/FRt1/detection_loss‚2.4816397375021224 1loss/SRqqt2/detection_loss‚0.8622701349904982 /loss/FBt2/detection_loss‚0.7827727430930702 %loss/total_loss‚45.47764841114193 0loss/FRt2/assignment_loss‚1.8039300322532654 0loss/FBt3/assignment_loss‚0.3089938682775634 1loss/FBt4/assignment_loss‚0.13925816919427234 % _timestamp‚1.7884213097150338e+09 2loss/SRqqt3/assignment_loss‚1.9885422552802734 2loss/SRqqt4/assignment_loss‚1.9263472440450087 .loss/FBt3/detection_loss‚2.632038871447245 2loss/SRqqt1/assignment_loss‚1.6441636482874553 1loss/SRqqt3/detection_loss‚1.9722343193633216 .loss/FBt4/detection_loss‚3.502335524559021 _runtime‚4780.464880213—«÷)Û¹‚(:… ÂÉäÔäì‘x gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q,Štº‚(:é ÂÉäÔŒ ¯z% disk./.usageGB‚1098.4156684875488 disk.dm-1.in‚7.3125 disk.sda.in‚ 7.36328125 disk.sda.out‚ 270.2109375 disk.sda3.in‚7.3125$ memory_percent‚96.93474769592285" proc.memory.rssMB‚ 2862.7578125 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625 network.recv‚ 1.707602e+06 disk.sda2.in‚0 disk.rbd2.out‚165.0625 disk.rbd4.out‚0 disk.rbd3.in‚0# disk.dm-0.out‚21170.28564453125 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd1.out‚ 2.39013671875 disk.sda3.out‚ 270.171875 disk.drbd1777.in‚ 1.03515625' proc.memory.availableMB‚ 502.2109375 disk.dm-1.out‚ 270.171875 disk.rbd0.out‚0 cpu‚25.298627467578253& disk.nvme0n1.out‚21170.28564453125 disk.rbd2.in‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd3.out‚0) proc.memory.percent‚17.47288703918457 network.sent‚ 1.1480674e+07) disk./.usagePercent‚73.71559714433221 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9765625 disk.dm-0.in‚ 4.9765625 disk.rbd0.in‚0 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚21170.28564453125ì Öª»‚(:… ÑÉäÔŸä¸x gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gïäz¼‚(:ï ÑÉäÔ˜Ö{ disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.sda.out‚ 270.97265625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9765625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625$ memory_percent‚96.93291187286377 network.recv‚ 1.714205e+06 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.dm-0.in‚ 4.9765625 disk.dm-1.in‚7.3125% disk./.usageGB‚1098.4159049987793 disk.nvme0c0n1.in‚ 4.9765625 disk.sda3.in‚7.3125 disk.rbd2.in‚0( proc.memory.availableMB‚ 502.51171875 cpu‚25.29789840358285 network.sent‚ 1.158979e+07) disk./.usagePercent‚73.71561301679934& disk.nvme0n1.out‚21170.55126953125 disk.sda3.out‚ 270.93359375 disk.rbd4.in‚0 disk.rbd4.out‚0) proc.memory.percent‚17.47288703918457 disk.rbd1.out‚2.453125# disk.dm-0.out‚21170.55126953125! disk.rbd2.out‚165.16845703125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚21170.55126953125 disk.dm-1.out‚ 270.93359375 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2862.7578125°$ô½‚(:… àÉäÔ½†Áx gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06“t¾‚(:é àÉäÔÍúîz$ memory_percent‚96.93448543548584! disk.dm-0.out‚21269.759765625 network.sent‚ 1.1595066e+07 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.out‚0 disk.sda.in‚ 7.36328125" proc.memory.rssMB‚ 2862.7578125) proc.memory.percent‚17.47288703918457 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.dm-1.out‚ 271.51171875 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 4.9765625% disk./.usageGB‚1098.4161682128906 disk.rbd4.in‚0& disk.nvme0c0n1.out‚21269.759765625 disk.sda.out‚ 271.55078125 proc.cpu.threads‚17$ disk.nvme0n1.out‚21269.759765625 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚2.453125 disk.sda2.in‚0( proc.memory.availableMB‚ 502.25390625( disk./.usagePercent‚73.7156306813192! disk.rbd2.out‚165.16845703125 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 271.51171875 disk.rbd1.in‚0 cpu‚25.29713307112833 network.recv‚ 1.717551e+06 disk.rbd4.out‚0 disk.dm-0.in‚ 4.9765625 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125G £e2¿- _wandbruntime‚4825  _runtime‚4825Ë#nnÆÀÀ & lr-AdamW/pg1‚0.0005851730141498617 & lr-AdamW/pg2‚0.0005851730141498617 trainer/global_step‚4999 $ _timestamp‚1.788421354524887e+09 _runtime‚4825.274960594 _step‚408sÒÅËÁÅ & lr-AdamW/pg1‚0.0005851730141498617 & lr-AdamW/pg2‚0.0005851730141498617 trainer/global_step‚4999 $ _timestamp‚1.788421354524887e+09 _runtime‚4825.274960594 _step‚408˜4ç2f2Â- _wandbruntime‚4825  _runtime‚4825•`³Qß'ÃÙO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207582105626624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9287859233792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198294264395776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198294246392832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9452057383936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222568054 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207582105626624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9260896902408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92371027 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130197027 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207582351833088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212771575958528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9288087437312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130171357 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33680453 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92371027 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33679813 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92371025 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚187763252970496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9261120551556 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚409 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 _runtime‚4825.276165113 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130171357 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9287859233792 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚197024373522052 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130197027 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212771552018432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54045198 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198294264395776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚197024132833544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚187763235931136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚222223609402368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4999k3Þ›ä'ÄÞO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9261120551556 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚409 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 _runtime‚4825.276165113 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130171357 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9287859233792 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚197024373522052 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130197027 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212771552018432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54045198 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198294264395776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚197024132833544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚187763235931136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚222223609402368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4999 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207582105626624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9287859233792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198294264395776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198294246392832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9452057383936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222568054 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207582105626624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9260896902408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92371027 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130197027 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207582351833088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212771575958528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9288087437312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130171357 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33680453 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92371027 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33679813 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92371025 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚187763252970496™¿:øJ2Å- _wandbruntime‚4826  _runtime‚4826ë‰v½ ÆãM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222585652 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9289925145088 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222611350 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54055555 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198332602339840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9262952951364 trainer/global_step‚4999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130196908 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222611350 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚187799555428352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92388744 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130222604 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92388746 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207622527484928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130222604 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198332584336896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9289925145088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87742353 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92388744 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87741708 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207622281265152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54055553 _runtime‚4826.143970092 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222585652 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212812721714688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198332602339840 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9289696928256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 _step‚410 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207622527484928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9289696928256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚222266960801792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197062267691000 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚197062508379716 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 % _timestamp‚1.7884213553933299e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9453937950208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33686798 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚212812745654784ÝÀÙ%î&ÇèM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207622527484928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130222604 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198332584336896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9289925145088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87742353 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92388744 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87741708 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207622281265152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54055553 _runtime‚4826.143970092 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222585652 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212812721714688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198332602339840 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9289696928256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 _step‚410 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207622527484928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9289696928256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚222266960801792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197062267691000 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚197062508379716 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 % _timestamp‚1.7884213553933299e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9453937950208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33686798 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚212812745654784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222585652 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9289925145088 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222611350 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54055555 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198332602339840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9262952951364 trainer/global_step‚4999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130196908 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222611350 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚187799555428352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92388744 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130222604 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92388746 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624š=δ42È- _wandbruntime‚4826  _runtime‚4826ñuí`ÉÚ /loss/FBt4/detection_loss‚1.9775840640068054 trainer/global_step‚4999 0loss/FRt1/assignment_loss‚3.8161864439646402 0loss/FRt3/assignment_loss‚3.5899700170213524 3loss/FBt3/assignment_loss‚0.0030253726290538907 1loss/FBt4/assignment_loss‚0.35819709665057337 /loss/FBt2/detection_loss‚0.5387702100307626 0loss/FBt1/assignment_loss‚0.5796745477287004 /loss/FRt3/detection_loss‚2.4832957454703073 .loss/FBt3/detection_loss‚4.080924252669017 2loss/SRqqt2/assignment_loss‚1.2589366709353862 1loss/SRqqt2/detection_loss‚0.9094451716193905 epoch‚4 _runtime‚4826.145047802 /loss/FRt4/assignment_loss‚3.404009348637349 2loss/SRqqt3/assignment_loss‚1.5916266636206553 1loss/FBt2/assignment_loss‚0.10533991889989347 _step‚411 /loss/FRt2/detection_loss‚1.6107528732882606 0loss/SRqqt1/detection_loss‚2.574045025743544 /loss/FBt1/detection_loss‚2.2352056801319122 0loss/FRt2/assignment_loss‚1.4280622707472908 2loss/SRqqt1/assignment_loss‚1.7457377244718373 2loss/SRqqt4/assignment_loss‚1.8222212825309148 1loss/SRqqt3/detection_loss‚1.9428685915011625 /loss/FRt1/detection_loss‚2.5108517891830866 .loss/FRt4/detection_loss‚2.480660422428234 1loss/SRqqt4/detection_loss‚2.1284422766078603 %loss/total_loss‚45.17583346051799 % _timestamp‚1.7884213553952072e+09IñíeÊß /loss/FRt1/detection_loss‚2.5108517891830866 .loss/FRt4/detection_loss‚2.480660422428234 1loss/SRqqt4/detection_loss‚2.1284422766078603 %loss/total_loss‚45.17583346051799 % _timestamp‚1.7884213553952072e+09 /loss/FBt4/detection_loss‚1.9775840640068054 trainer/global_step‚4999 0loss/FRt1/assignment_loss‚3.8161864439646402 0loss/FRt3/assignment_loss‚3.5899700170213524 3loss/FBt3/assignment_loss‚0.0030253726290538907 1loss/FBt4/assignment_loss‚0.35819709665057337 /loss/FBt2/detection_loss‚0.5387702100307626 0loss/FBt1/assignment_loss‚0.5796745477287004 /loss/FRt3/detection_loss‚2.4832957454703073 .loss/FBt3/detection_loss‚4.080924252669017 2loss/SRqqt2/assignment_loss‚1.2589366709353862 1loss/SRqqt2/detection_loss‚0.9094451716193905 epoch‚4 _runtime‚4826.145047802 /loss/FRt4/assignment_loss‚3.404009348637349 2loss/SRqqt3/assignment_loss‚1.5916266636206553 1loss/FBt2/assignment_loss‚0.10533991889989347 _step‚411 /loss/FRt2/detection_loss‚1.6107528732882606 0loss/SRqqt1/detection_loss‚2.574045025743544 /loss/FBt1/detection_loss‚2.2352056801319122 0loss/FRt2/assignment_loss‚1.4280622707472908 2loss/SRqqt1/assignment_loss‚1.7457377244718373 2loss/SRqqt4/assignment_loss‚1.8222212825309148 1loss/SRqqt3/detection_loss‚1.9428685915011625›Ç„Ë‚(:„ ïÉäÔìÍŒx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ€Ö^rÌ‚(:ç ïÉäÔõ»{ disk.rbd3.out‚0! disk.rbd2.out‚165.24658203125 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125# memory_percent‚96.9329833984375 network.sent‚ 1.1601234e+07 disk.rbd2.in‚0 disk.sda1.in‚ 0.05078125" proc.memory.rssMB‚ 2862.7578125 proc.cpu.threads‚17 disk.dm-0.in‚ 4.9765625 cpu‚25.296311957326708 disk.nvme0n1.in‚ 4.9765625! disk.dm-0.out‚21368.837890625 disk.sda2.out‚ 0.0390625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd4.out‚0! proc.memory.availableMB‚502.5) proc.memory.percent‚17.47288703918457 network.recv‚ 1.72218e+06% disk./.usageGB‚1098.4165954589844$ disk.nvme0n1.out‚21368.837890625 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625& disk.nvme0c0n1.out‚21368.837890625 disk.sda.out‚ 272.03515625 disk.sda3.in‚7.3125 disk.rbd1.in‚0 disk.rbd1.out‚ 2.51611328125 disk.sda2.in‚0 disk.dm-1.out‚ 271.99609375 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda3.out‚ 271.99609375) disk./.usagePercent‚73.71565935416304C¥w¾Í‚(:… þÉäÔÆÎ’x gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“‚á´~΂(:ó þÉäÔÊ–Ú{ disk.sda2.in‚0( proc.memory.availableMB‚ 502.26171875 network.recv‚ 1.72817e+06 disk.sda.out‚ 272.55078125 disk.rbd2.in‚0 disk.drbd1777.out‚0 cpu‚25.29555087933534 network.sent‚ 1.1709658e+07) disk./.usagePercent‚73.71565807412537$ disk./.usageGB‚1098.416576385498( disk.nvme0c0n1.out‚21468.70263671875 disk.sda3.out‚ 272.51171875& disk.nvme0n1.out‚21468.70263671875# disk.dm-0.out‚21468.70263671875 disk.dm-1.in‚7.3125 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 4.9765625 disk.dm-1.out‚ 272.51171875$ memory_percent‚96.93443775177002 disk.sda.in‚ 7.36328125 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 2.51611328125) proc.memory.percent‚17.47288703918457 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.nvme0n1.in‚ 4.9765625! disk.rbd2.out‚166.82080078125 disk.rbd0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2862.7578125 disk.rbd4.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚ 1.03515625 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625ä©Ú ›Ï‚(: ÊäÔ™ñx gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.467' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.293400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:a qyЂ(:î ÊäÔ¿®î{ disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125$ memory_percent‚97.95503616333008 disk.rbd3.out‚0 disk.dm-0.in‚ 4.9765625! disk.dm-0.out‚21468.994140625 disk.nvme0n1.in‚ 4.9765625 disk.rbd0.in‚0 disk.sda.out‚ 272.99609375 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 272.95703125 network.sent‚ 1.1716092e+07% disk./.usageGB‚1098.4161491394043 disk.rbd1.out‚ 2.51611328125 disk.rbd2.in‚0 disk.sda3.in‚7.3125) proc.memory.percent‚17.47288703918457 cpu‚25.294795458231956 network.recv‚ 1.733193e+06) disk./.usagePercent‚73.71562940128152 disk.rbd1.in‚0 disk.rbd3.in‚0& proc.memory.availableMB‚ 335.046875! disk.rbd2.out‚166.83642578125 disk.drbd1777.out‚0& disk.nvme0c0n1.out‚21468.994140625" proc.memory.rssMB‚ 2862.7578125 disk.dm-1.in‚7.3125$ disk.nvme0n1.out‚21468.994140625 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.sda3.out‚ 272.95703125 proc.cpu.threads‚17À 2Ñ- _wandbruntime‚4871  _runtime‚4871õü…ÇÒÁ & lr-AdamW/pg1‚0.0005800080306059842 & lr-AdamW/pg2‚0.0005800080306059842 trainer/global_step‚5049 % _timestamp‚1.7884214005381114e+09 _runtime‚4871.288084805 _step‚412ìq¿ÌÓÆ _step‚412 & lr-AdamW/pg1‚0.0005800080306059842 & lr-AdamW/pg2‚0.0005800080306059842 trainer/global_step‚5049 % _timestamp‚1.7884214005381114e+09 _runtime‚4871.288084805œ‰kÒ2Ô- _wandbruntime‚4871  _runtime‚4871±ì‹«ÕÙO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54562950 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93256975 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9545837004800 _step‚413 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚209591148173312 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚200211161598976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚200211143596032 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131452259 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224709234 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9379986574336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54562948 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚4871.289053509 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93256975 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34002274 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214830060771328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34001594 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224375590334464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 283501568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209590901966848 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88564542 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214830036831232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198930889297552 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131452259 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131477929 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88565224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307441664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131477929 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224734906 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G>DeviceStatsMonitor.on_train_batc*G4 h_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209590901966848 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9379758370816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚200211161598976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93256977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚189578375863296 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9352530473616 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 trainer/global_step‚5049 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209590901966848 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9379758370816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚200211161598976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93256977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚189578375863296 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9352530473616 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 trainer/global_step‚5049 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88565224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307441664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0…âR®2×- _wandbruntime‚4872  _runtime‚4872ÁO¬æ&ØàM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131477810 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224778202 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131503506 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9354586522572 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚198969024155008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34008721 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34008073 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788421401417273e+09 _runtime‚ 4872.1676245 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209631323825152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131477810 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 _step‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93274696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88581376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚189614661281792 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93274696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214871206527488 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚200249481540096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214871230467584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54573303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚224418942258176 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚189614678321152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224752504 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224752504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9547711790592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93274694 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9381824282112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650|r‹­%ÙåM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214871206527488 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚200249481540096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214871230467584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54573303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚224418942258176 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚189614678321152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224752504 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224752504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9547711790592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93274694 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9381824282112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131477810 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224778202 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131503506 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9354586522572 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚198969024155008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34008721 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34008073 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788421401417273e+09 _runtime‚ 4872.1676245 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209631323825152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131477810 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 _step‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93274696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88581376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚189614661281792 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93274696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DevicežBêÒ>StatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211599698307072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94142927 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9444164044824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202128058802176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚202128040799232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94142925 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34323194 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚202128040799232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226901758 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚665 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89404557 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9471885711360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132758831 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226527829335552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55080700 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307441664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9471885711360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132758831 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226527829335552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55080700 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307441664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211599698307072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94142927 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9444164044824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202128058802176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚202128040799232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94142925 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34323194 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896¡"æ{é2é- _wandbruntime‚4918  _runtime‚4918Fƒ*VWêãM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34330234 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚200875780619016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55091053 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚226570560747008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89420657 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9473495202304 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚191429784174592 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 trainer/global_step‚5099 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132758712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202166378743296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9445996444424 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9473723419136 O&Ùo°’ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9473723419136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34329604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226570857689600 _step‚418 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚200876021307732 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226945054 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226919356 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55091055 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9640869406720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚191429801213952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚211640120165376 % _timestamp‚1.7884214473656733e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚216929715280384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226945054 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚4918.116204404 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9641142409216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94160644 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9473495202304 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚202166396746240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94160646 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226919356 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211639873945600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202166378743296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94160646 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94160644 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132758712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚216929691340288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673I¡Ë>î&ëèM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226945054 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚4918.116204404 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9641142409216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94160644 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9473495202304 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚202166396746240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94160646 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226919356 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211639873945600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202166378743296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94160646 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94160644 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132758712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚216929691340288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34330234 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚200875780619016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55091053 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚226570560747008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89420657 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9473495202304 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚191429784174592 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 trainer/global_step‚5099 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132758712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202166378743296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9445996444424 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9473723419136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9473723419136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34329604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226570857689600 _step‚418 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚200876021307732 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226945054 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226919356 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55091055 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9640869406720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚191429801213952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚211640120165376 % _timestamp‚1.7884214473656733e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚216929715280384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2¢¾Ð62ì- _wandbruntime‚4918  _runtime‚4918{i©Ã_íÙ 3loss/FBt3/assignment_loss‚0.0067654711892828345 1loss/SRqqt2/detection_loss‚0.9782754914605454 /loss/FRt2/detection_loss‚1.8605583360337692 2loss/SRqqt2/assignment_loss‚1.1737949020956477 /loss/FRt3/detection_loss‚2.4282483953110714 %loss/total_loss‚45.33920689782193 % _timestamp‚1.7884214473674834e+09 1loss/SRqqt1/detection_loss‚2.4634489193558693 epoch‚4 trainer/global_step‚5099 1loss/FBt4/assignment_loss‚0.23351969132199885 .loss/FRt1/detection_loss‚2.466728223931222 0loss/FRt3/assignment_loss‚3.1977189259326204 /loss/FBt1/detection_loss‚2.8679721781185696 _runtime‚4918.117284869 0loss/FRt4/assignment_loss‚3.4430401707306886 1loss/SRqqt4/assignment_loss‚1.794511569974323 1loss/FBt2/assignment_loss‚0.08672990344330936 /loss/FRt4/detection_loss‚2.4281665736284013 .loss/FBt3/detection_loss‚2.588992141187191 .loss/FBt4/detection_loss‚2.988556170463562 2loss/SRqqt1/assignment_loss‚2.0696456050500274 /loss/FBt2/detection_loss‚0.6041195731861203 _step‚419 /loss/FRt1/assignment_loss‚4.384827979973385 0loss/FRt2/assignment_loss‚1.4186074500498564 1loss/FBt1/assignment_loss‚0.23868153119526272 0loss/SRqqt3/detection_loss‚1.834318539925984 1loss/SRqqt4/detection_loss‚1.8582665249705315 2loss/SRqqt3/assignment_loss‚1.9237126292926925Õ"5\dîÞ epoch‚4 trainer/global_step‚5099 1loss/FBt4/assignment_loss‚0.23351969132199885 .loss/FRt1/detection_loss‚2.466728223931222 0loss/FRt3/assignment_loss‚3.1977189259326204 /loss/FBt1/detection_loss‚2.8679721781185696 _runtime‚4918.117284869 0loss/FRt4/assignment_loss‚3.4430401707306886 1loss/SRqqt4/assignment_loss‚1.794511569974323 1loss/FBt2/assignment_loss‚0.08672990344330936 /loss/FRt4/detection_loss‚2.4281665736284013 .loss/FBt3/detection_loss‚2.588992141187191 .loss/FBt4/detection_loss‚2.988556170463562 2loss/SRqqt1/assignment_loss‚2.0696456050500274 /loss/FBt2/detection_loss‚0.6041195731861203 _step‚419 /loss/FRt1/assignment_loss‚4.384827979973385 0loss/FRt2/assignment_loss‚1.4186074500498564 1loss/FBt1/assignment_loss‚0.23868153119526272 0loss/SRqqt3/detection_loss‚1.834318539925984 1loss/SRqqt4/detection_loss‚1.8582665249705315 2loss/SRqqt3/assignment_loss‚1.9237126292926925 3loss/FBt3/assignment_loss‚0.0067654711892828345 1loss/SRqqt2/detection_loss‚0.9782754914605454 /loss/FRt2/detection_loss‚1.8605583360337692 2loss/SRqqt2/assignment_loss‚1.1737949020956477 /loss/FRt3/detection_loss‚2.4282483953110714 %loss/total_loss‚45.33920689782193 % _timestamp‚1.7884214473674834e+09 1loss/SRqqt1/detection_loss‚2.4634489193558693£^Ÿ6›ï‚(: ÉÊäÔ±ª¶x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.859' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.971799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228679751369728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95028875 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34643959 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚204044938002432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213608740853760 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213608494647296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213608494647296 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229068610 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193208621648896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90243063 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134014063 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134014063 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9563784848384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134039733 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204044956005376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9563556644864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55598450 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204044938002432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9563556644864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202744402225568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚218947030396928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95028877 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 trainer/global_step‚5149 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213608740853760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55598448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34644613 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9536021265180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90242407 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218947006456832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚193208604609536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocatÉ<‘ãÛed‚262 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228679452316672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 _runtime‚4963.687068465 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202744642914076 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134039733û§Á+ä'úÞO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213608494647296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213608494647296 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229068610 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193208621648896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90243063 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134014063 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134014063 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9563784848384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134039733 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204044956005376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9563556644864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55598450 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204044938002432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9563556644864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202744402225568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚218947030396928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95028877 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 trainer/global_step‚5149 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213608740853760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55598448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34644613 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9536021265180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90242407 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218947006456832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚193208604609536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228679452316672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 _runtime‚4963.687068465 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202744642914076 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134039733 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚421 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95028877 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228679751369728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95028875 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34643959 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚204044938002432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213608740853760 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824¥g#H2û- _wandbruntime‚4964  _runtime‚4964WTIé&üãM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229111906 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95046596 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚204083275946496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202782777771740 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134065310 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9734308633088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90259475 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193244924106752 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229086208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9565394339328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34651351 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204083293949440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9565622556160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229086208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55608803 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 % _timestamp‚1.7884214938044157e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229111906 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95046594 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204083293949440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228722783825920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 _step‚422 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213648670285824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218988176153088 _runtime‚4964.554949325 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213648916505600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34650672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9537630015632 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204083275946496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218988200093184 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202782537083024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193244907067392 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213648916505600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 trainer/global_step‚5149 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134065310 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9734308633088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90259475 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193244924106752 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229086208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9565394339328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34651351 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204083293949440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9565622556160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229086208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55608803 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 % _timestamp‚1.7884214938044157e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229111906 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95046594 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204083293949440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228722783825920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 _step‚422 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213648670285824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218988176153088 _runtime‚4964.554949325 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213648916505600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34650672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9537630015632 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204083275946496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218988200093184 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¦ûšˆ—2þ- _wandbruntime‚4964  _runtime‚4964>VßÕÿÕ 2loss/SRqqt1/assignment_loss‚2.3039205652825974 0loss/FRt4/assignment_loss‚3.8086018373326556 /loss/FRt1/detection_loss‚2.3567063751129003 /loss/FRt3/detection_loss‚2.3716327002707946 /loss/FBt1/detection_loss‚2.6134411295255027 _runtime‚4964.555913662 0loss/FBt4/assignment_loss‚0.2316217323144277 /loss/FRt2/assignment_loss‚1.744268102305276 2loss/SRqqt2/assignment_loss‚1.1665718627886639 1loss/FBt2/assignment_loss‚0.16091934791328644 0loss/FBt3/assignment_loss‚0.3517512749143255 /loss/FRt4/detection_loss‚2.3267266445043715 0loss/SRqqt1/detection_loss‚2.577103556079023 /loss/FBt3/detection_loss‚2.7779011726379395 2loss/SRqqt4/assignment_loss‚1.7992496318659004 1loss/SRqqt2/detection_loss‚0.8814255124256934 0loss/SRqqt4/detection_loss‚1.899425560114335 0loss/FBt1/assignment_loss‚0.4464145190237711 /loss/FBt2/detection_loss‚0.6593482990247699 epoch‚4 % _timestamp‚1.7884214938060Q¯ý«†699e+09 /loss/FRt1/assignment_loss‚4.094665056619888 /loss/FRt3/assignment_loss‚3.552180389140515 1loss/SRqqt3/detection_loss‚1.5651412482919365 _step‚423 2loss/SRqqt3/assignment_loss‚2.0131326633794555 /loss/FRt2/detection_loss‚1.5471038908070447 .loss/FBt4/detection_loss‚4.697828114032745 &loss/total_loss‚47.947081185707816 trainer/global_step‚5149<¸Œ_`€Ú 0loss/FRt4/assignment_loss‚3.8086018373326556 /loss/FRt1/detection_loss‚2.3567063751129003 /loss/FRt3/detection_loss‚2.3716327002707946 /loss/FBt1/detection_loss‚2.6134411295255027 _runtime‚4964.555913662 0loss/FBt4/assignment_loss‚0.2316217323144277 /loss/FRt2/assignment_loss‚1.744268102305276 2loss/SRqqt2/assignment_loss‚1.1665718627886639 1loss/FBt2/assignment_loss‚0.16091934791328644 0loss/FBt3/assignment_loss‚0.3517512749143255 /loss/FRt4/detection_loss‚2.3267266445043715 0loss/SRqqt1/detection_loss‚2.577103556079023 /loss/FBt3/detection_loss‚2.7779011726379395 2loss/SRqqt4/assignment_loss‚1.7992496318659004 1loss/SRqqt2/detection_loss‚0.8814255124256934 0loss/SRqqt4/detection_loss‚1.899425560114335 0loss/FBt1/assignment_loss‚0.4464145190237711 /loss/FBt2/detection_loss‚0.6593482990247699 epoch‚4 % _timestamp‚1.7884214938060699e+09 /loss/FRt1/assignment_loss‚4.094665056619888 /loss/FRt3/assignment_loss‚3.552180389140515 1loss/SRqqt3/detection_loss‚1.5651412482919365 _step‚423 2loss/SRqqt3/assignment_loss‚2.0131326633794555 /loss/FRt2/detection_loss‚1.5471038908070447 .loss/FBt4/detection_loss‚4.697828114032745 &loss/total_loss‚47.947081185707816 trainer/global_step‚5149 2loss/SRqqt1/assignment_loss‚2.3039205652825974§ý ¯yš‚(: öÊäÔ°¾ x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚92.465' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.493000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07~>ôo‚‚(:ä öÊäÔƒ£Š{% disk./.usageGB‚1098.4158363342285 network.sent‚ 1.1962517e+07 disk.sda1.out‚0 disk.sda3.out‚ 276.7421875 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 276.7421875 disk.rbd2.out‚168.7978515625 disk.rbd3.out‚0 disk.dm-0.in‚ 4.9765625 disk.dm-0.out‚ 21966.1796875 disk.rbd4.out‚0$ memory_percent‚97.94495105743408" proc.memory.rssMB‚ 2862.7578125 cpu‚25.289553930910934 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚ 4.9765625 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd0.in‚0 network.recv‚ 1.765846e+06) disk./.usagePercent‚73.71560840866373 disk.sda.out‚ 276.78125 disk.rbd1.in‚0 disk.rbd1.out‚ 2.51611328125 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 21966.1796875 disk.drbd1777.out‚0 disk.sda3.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.rbd3.in‚0" disk.nvme0n1.out‚ 21966.1796875 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 4.9765625( proc.memory.availableMB‚ 336.69921875 disk.sda1.in‚ 0.05078125) proc.memory.percent‚17.47288703918457 proc.cpu.threads‚17Õýe ƒ‚(:… …ËäÔ±ÛŸx gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.027' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0꣆w„‚(:ì …ËäÔÊÙÅz disk.rbd0.out‚0 disk.rbd4.in‚0 cpu‚25.288841793562018 disk.rbd2.out‚168.7978515625 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125# disk.nvme0n1.out‚22065.28515625 disk.dm-1.out‚ 277.36328125 disk.drbd1777.in‚ 1.03515625 network.sent‚ 1.2071402e+07 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.drbd1777.out‚0) proc.memory.percent‚17.47288703918457% disk.nvme0c0n1.out‚22065.28515625 disk.dm-0.in‚ 4.9765625 disk.dm-0.out‚22065.28515625 disk.rbd4.out‚0$ memory_percent‚97.94490337371826 disk.sda.out‚ 277.40234375( proc.memory.availableMB‚ 336.70703125 proc.cpu.threads‚17 disk.rbd1.out‚ 2.51611328125 disk.sda2.out‚ 0.0390625" proc.memory.rssMB‚ 2862.7578125 disk.sda3.out‚ 277.36328125) disk./.usagePercent‚73.71560840866373 disk.sda3.in‚7.3125 disk.nvme0n1.in‚ 4.9765625 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 network.recv‚ 1.772792e+06% disk./.usageGB‚1098.4158363342285 disk.dm-1.in‚7.3125 disk.rbd0.in‚0óeÎÒ…‚(:… ”ËäÔÔÔžx gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.654' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[¡­­†‚(:ö ”ËäÔùâz network.sent‚ 1.2076823e+07) disk./.usagePercent‚73.71584854373083 disk.dm-1.out‚ 278.03515625 cpu‚25.288126003109042 disk.rbd0.in‚0 disk.sda.out‚ 278.07421875) proc.memory.percent‚17.47288703918457 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.dm-0.in‚ 4.9765625 disk.drbd1777.out‚0 network.recv‚ 1.776495e+06! disk.rbd2.out‚168.90771484375 disk.rbd4.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚7.3125( disk.nvme0c0n1.out‚22168.53564453125 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚97.94495105743408% disk./.usageGB‚1098.4194145202637 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 278.03515625 disk.rbd1.out‚ 2.51611328125# disk.dm-0.out‚22168.53564453125 disk.sda2.out‚ 0.0390625 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125 disk.nvme0n1.in‚ 4.9765625& disk.nvme0n1.out‚22168.53564453125( proc.memory.availableMB‚ 336.69921875" proc.memory.rssMB‚ 2862.7578125u©2‡- _wandbruntime‚5009  _runtime‚5009KºŽÇˆÁ trainer/global_step‚5199 % _timestamp‚1.7884215392168908e+09 _runtime‚5009.966950849 _step‚424 & lr-AdamW/pg1‚0.0005642937932847355 & lr-AdamW/pg2‚0.00056429379328473554p˜Ì‰Æ _runtime‚5009.966950849 _step‚424 & lr-AdamW/pg1‚0.0005642937932847355 & lr-AdamW/pg2‚0.0005642937932847355 trainer/global_step‚5199 % _timestamp‚1.7884215392168908e+09¨÷õåm2Š- _wandbruntime‚5009  _runtime‚5009Ô²ß'‹ÙO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚215617290987520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚205961853208576 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95914825 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230831282176000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 _runtime‚5009.967962004 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135320635 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135294965 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204651399378084 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34967676 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9655683985408 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230831589617664 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215617537193984 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚205961853208576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135294965 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚5199 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9655455781888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9627431187240 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚205961835205632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135320635 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231235462 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚221005491269632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215617290987520 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221005515209728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231209790 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9655683985408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91084564 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚195023744541696 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690Áµ´eä'ŒÞO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231235462 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚221005491269632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215617290987520 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221005515209728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231209790 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9655683985408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91084564 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚195023744541696 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚215617290987520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚205961853208576 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95914825 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230831282176000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 _runtime‚5009.967962004 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135320635 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135294965 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204651399378084 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34967676 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9655683985408 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230831589617664 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215617537193984 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚205961853208576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135294965 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚5199 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9655455781888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9627431187240 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚205961835205632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135320635©i5O™‚(:‚ £ËäÔç”x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.32' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.064 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐÌaèwŽ‚(:ì £ËäÔÖæÆz disk.drbd1777.out‚0 disk.rbd4.in‚0$ memory_percent‚97.94645309448242 disk.rbd3.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 278.46875 disk.rbd0.out‚0 network.sent‚ 1.2082886e+07) disk./.usagePercent‚73.71585673597193 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125" proc.memory.rssMB‚ 2862.7578125 network.recv‚ 1.781047e+06 disk.sda1.in‚ 0.05078125 disk.rbd1.in‚0& disk.nvme0n1.out‚22267.63037109375 disk.sda2.in‚0! disk.rbd2.out‚170.41943359375 disk.rbd1.out‚ 2.51611328125 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 4.9765625 cpu‚25.28741906782214 disk.dm-1.in‚7.3125 proc.cpu.threads‚17% disk./.usageGB‚1098.4195365905762 disk.rbd3.out‚0 disk.sda3.out‚ 278.46875 disk.nvme0n1.in‚ 4.9765625# disk.dm-0.out‚22267.63037109375 disk.rbd4.out‚0( disk.nvme0c0n1.out‚22267.63037109375 disk.drbd1777.in‚ 1.03515625 disk.dm-0.in‚ 4.9765625& proc.memory.availableMB‚ 336.453125) proc.memory.percent‚17.47288703918457 disk.sda1.out‚0 disk.sda.out‚ 278.5078125 disk.rbd0.in‚0ä‹ïµ2- _wandbruntime‚5010  _runtime‚5010†›.ƒ‹ãM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9657293476352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884215401228998e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221046684905984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9827931063808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231278758 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95932546 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91100838 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206000191152640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206000173149696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 trainer/global_step‚5199 _step‚426 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9827655964160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚195060029960192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215657466626048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215657712845824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221046660965888 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚349(^74930 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204689293547032 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚195060046999552 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135320516 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9629263586840 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135320516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231253060 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206000173149696 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9657293476352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9629487236196 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34974285 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231253060 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230874615969792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚230874316930048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚204689534235748 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95932544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215657712845824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91101485 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9657521693184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231278758 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91100838 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206000191152640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206000173149696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 trainer/global_step‚5199 _step‚426 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9827655964160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚195060029960192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215657466626048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215657712845824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221046660965888 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34974930 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204689293547032 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚195060046999552 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135320516 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9629263586840 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135320516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231253060 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206000173149696 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9657293476352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9629487236196 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34974285 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231253060 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230874615969792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚230874316930048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚204689534235748 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95932544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215657712845824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91101485 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9657521693184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231278758 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221046684905984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9827931063808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231278758 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299ª£S2’- _wandbruntime‚5010  _runtime‚5010 ¤'ý]“× /loss/FRt4/assignment_loss‚4.018553808678028 1loss/FBt2/assignment_loss‚0.14575861611092883 0loss/FBt3/assignment_loss‚0.2483511786122108 /loss/FBt1/detection_loss‚2.7420708409377506 &loss/total_loss‚45.649156951047836 epoch‚4 2loss/SRqqt4/assignment_loss‚1.5681795920519268 .loss/FRt1/detection_loss‚2.320976122483319 .loss/FRt3/detection_loss‚2.497969220081965 /loss/FBt2/detection_loss‚0.7031940055720798 % _timestamp‚1.7884215401251268e+09 0loss/FRt1/assignment_loss‚3.8021155432158826 /loss/FRt3/assignment_loss‚3.606486584459032 2loss/SRqqt2/assignment_loss‚1.0907875215013823 .loss/FRt4/detection_loss‚2.491423559050227 1loss/SRqqt4/detection_loss‚1.8641404173537797 /loss/FBt3/detection_loss‚2.0551124289631844 _runtime‚5010.874975799 2loss/SRqqt1/assignment_loss‚2.4643254327774047 1loss/FBt1/assignment_loss‚0.22069775299834354 1loss/FBt4/assignment_loss‚0.17881139347122776 1loss/SRqqt3/detection_loss‚1.6523553211580624 0loss/FRt2/assignment_loss‚1.9739456300077767 2loss/SRqqt3/assignment_loss‚1.6701798668300563 _step‚427 /loss/FRt2/detection_loss‚2.2020941121054105 1loss/SRqqt1/detection_loss‚2.4497126269340517 1loss/SRqqt2/detection_loss‚1.1588448151325186 .loss/FBt4/detection_loss‚2.523070560561286 trainer/global_step‚5199€Cªb”Ü _step‚427 /loss/FRt2/detection_loss‚2.2020941121054105 1loss/SRqqt1/detection_loss‚2.4497126269340517 1loss/SRqqt2/detection_loss‚1.1588448151325186 .loss/FBt4/detection_loss‚2.523070560561286 trainer/global_step‚5199 /loss/FRt4/assignment_loss‚4.018553808678028 1loss/FBt2/assignment_loss‚0.14575861611092883 0loss/FBt3/assignment_loss‚0.2483511786122108 /loss/FBt1/detection_loss‚2.7420708409377506 &loss/total_loss‚45.649156951047836 epoch‚4 2loss/SRqqt4/assignment_loss‚1.5681795920519268 .loss/FRt1/detection_loss‚2.320976122483319 .loss/FRt3/detection_loss‚2.497969220081965 /loss/FBt2/detection_loss‚0.7031940055720798 % _timestamp‚1.7884215401251268e+09 0loss/FRt1/assignment_loss‚3.8021155432158826 /loss/FRt3/assignment_loss‚3.606486584459032 2loss/SRqqt2/assignment_loss‚1.0907875215013823 .loss/FRt4/detection_loss‚2.491423559050227 1loss/SRqqt4/detection_loss‚1.8641404173537797 /loss/FBt3/detection_loss‚2.0551124289631844 _runtime‚5010.874975799 2loss/SRqqt1/assignment_loss‚2.4643254327774047 1loss/FBt1/assignment_loss‚0.22069775299834354 1loss/FBt4/assignment_loss‚0.17881139347122776 1loss/SRqqt3/detection_loss‚1.6523553211580624 0loss/FRt2/assignment_loss‚1.9739456300077767 2loss/SRqqt3/assignment_loss‚1.6701798668300563«‰¢2•‚(:… ²ËäÔ¿ë¯x gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ^F®j–‚(:ß ²ËäÔãÈá{ disk.drbd1777.out‚0 disk.rbd3.out‚0) disk./.usagePercent‚73.71585673597193 disk.sda.in‚ 7.36328125 proc.cpu.threads‚17 network.sent‚ 1.2180025e+07 disk.sda.out‚ 279.140625 disk.rbd1.in‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 cpu‚25.28675101596925% disk./.usageGB‚1098.4195365905762 disk.nvme0n1.in‚ 4.9765625 disk.dm-0.in‚ 4.9765625 disk.dm-1.out‚ 279.1015625 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9765625 network.recv‚ 1.786714e+06 disk.sda3.in‚7.3125 disk.rbd1.out‚ 2.51611328125 disk.dm-0.out‚ 22268.15625 disk.rbd0.out‚0$ memory_percent‚97.94437885284424( proc.memory.availableMB‚ 336.79296875) proc.memory.percent‚17.47288703918457 disk.dm-1.in‚7.3125 disk.rbd2.in‚0! disk.rbd2.out‚170.49755859375 disk.rbd0.in‚0 disk.sda3.out‚ 279.1015625 disk.nvme0n1.out‚ 22268.15625 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0" disk.nvme0c0n1.out‚ 22268.15625 disk.rbd3.in‚0" proc.memory.rssMB‚ 2862.7578125FHO–—‚(:… ÁËäÔ“ŸÇx gpu.0.gpu‚18.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚104.281' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8562 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûmzr˜‚(:ç ÁËäÔéÍz disk.sda2.out‚ 0.0390625 cpu‚25.286012814165105 network.sent‚ 1.2278374e+07 disk.sda3.in‚7.3125# disk.nvme0n1.out‚22367.19921875 disk.dm-0.out‚22367.19921875 disk.dm-1.out‚ 279.6484375) disk./.usagePercent‚73.71583753540688 disk.sda.in‚ 7.36328125 disk.dm-1.in‚7.3125 disk.nvme0c0n1.in‚ 4.9765625% disk./.usageGB‚1098.4192504882812 disk.sda3.out‚ 279.6484375 disk.dm-0.in‚ 4.9765625 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2862.7578125 network.recv‚ 1.792889e+06 disk.rbd3.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚170.49755859375 disk.rbd4.in‚0$ memory_percent‚97.94595241546631( proc.memory.availableMB‚ 336.53515625 disk.nvme0n1.in‚ 4.9765625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125 proc.cpu.threads‚17 disk.sda.out‚279.6875% disk.nvme0c0n1.out‚22367.19921875) proc.memory.percent‚17.47288703918457 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.51611328125 disk.sda2.in‚0å‡<ðŽ™‚(:ƒ ÐËäÔËùz gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚97.095' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™õc^tš‚(:é ÐËäÔä—¸|" proc.memory.rssMB‚ 2862.7578125 cpu‚25.285297034455755) disk./.usagePercent‚73.71585033578357! disk.dm-0.out‚22466.587890625 disk.rbd2.out‚ 170.669921875 disk.sda1.out‚0$ disk.nvme0n1.out‚22466.587890625 disk.dm-0.in‚ 4.9765625 disk.rbd0.out‚0 disk.sda.out‚ 280.28125( proc.memory.availableMB‚ 336.82421875 disk.rbd1.out‚ 2.51611328125 disk.nvme0n1.in‚ 4.9765625 disk.dm-1.out‚ 280.2421875& disk.nvme0c0n1.out‚22466.587890625 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 280.2421875% disk./.usageGB‚1098.4194412231445 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚7.3125 proc.cpu.threads‚17 network.recv‚ 1.795986e+06 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.drbd1777.out‚0$ memory_percent‚97.94418811798096) proc.memory.percent‚17.47288703918457 network.sent‚ 1.2283743e+07 disk.sda.in‚ 7.36328125Ivf2›- _wandbruntime‚5056  _runtime‚50567Ù­ÛÇœÁ & lr-AdamW/pg2‚0.0005589862573325083 trainer/global_step‚5249 % _timestamp‚1.7884215857305121e+09 _runtime‚5056.480403283 _step‚428 & lr-AdamW/pg1‚0.0005589862573325083©4ÌÆ & lr-AdamW/pg2‚0.0005589862573325083 trainer/global_step‚5249 % _timestamp‚1.7884215857305121e+09 _runtime‚5056.480403283 _step‚428 & lr-AdamW/pg1‚0.0005589862573325083¬Õݹ2ž- _wandbruntime‚5056  _runtime‚5056›i­^!ŸØO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9747583122432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206558155842092 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 $ _timestamp‚1.788421585731182e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136575867 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136601537 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚223063976082432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚207878750411776 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232983245096960 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚429 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9747583122432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚217626333534208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚207878732408832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚217626087327744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207878750411776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚206557915153584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9719064758448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚232983546247168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207878732408832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196838850395136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9747354918912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91926952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35292332 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96800777 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233402314 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚5249 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217626087327744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56633948 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 _runtime‚5056.481287259 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9747354918912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 =7DeviceStatsMoWkÕ_€nitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96800777 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136601537 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196838867434496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233376642 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96800775 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428äV«ã' ÝO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136575867 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 $ _timestamp‚1.788421585731182e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136575867 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136601537 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚223063976082432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚207878750411776 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232983245096960 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚429 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9747583122432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚217626333534208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚207878732408832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚217626087327744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207878750411776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚206557915153584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9719064758448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚232983546247168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207878732408832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196838850395136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9747354918912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91926952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35292332 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96800777 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233402314 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚5249 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217626087327744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56633948 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 _runtime‚5056.481287259 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9747354918912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96800777 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136601537 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196838867434496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233376642 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96800775 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9747583122432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206558155842092 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884215865938225e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233419912 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9749192613376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96818496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136601418 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206596050011040 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196875169892352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233445610 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9721120807404 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚217666509186048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9749420830208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _step‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 283488256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚207917070352896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 _runtime‚5057.344436859 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136601418 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207917088355840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207917088355840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35299422 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9921410311680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233419912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196875152852992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91943727 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9720897158048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91943084 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚223105169718784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56644303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚233026580030464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96818494 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233026272602112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233445610 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136627114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _step‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 283488256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚207917070352896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 _runtime‚5057.344436859 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136601418 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207917088355840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207917088355840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35299422 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9921410311680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233419912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196875152852992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91943727 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9720897158048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91943084 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚223105169718784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56644303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚233026580030464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96818494 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233026272602112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233445610 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136627114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884215865938225e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233419912 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9749192613376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96818496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136601418 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206596050011040 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196875169892352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233445610 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299®ºÝ*½2¤- _wandbruntime‚5057  _runtime‚5057àÁP1¥Ö 1loss/SRqqt3/detection_loss‚1.9230529049347187 .loss/FBt3/detection_loss‚3.974043369293213 _step‚431 0loss/FRt4/assignment_loss‚3.2491766621426836 2loss/SRqqt1/assignment_loss‚1.6438105429212253 0loss/FBt1/assignment_loss‚0.6991546335112717 /loss/FRt1/detection_loss‚2.3143215455660004 /loss/FRt3/detection_loss‚2.5566875880414788 epoch‚4 1loss/SRqqt4/assignment_loss‚1.456762635814292 1loss/FBt2/assignment_loss‚0.17165244916967026 0loss/FBt3/assignment_loss‚0.1323621767029787 1loss/FBt4/assignment_loss‚0.07428035450478394 0loss/SRqqt1/detection_loss‚2.088570772182374 %loss/total_loss‚44.45386027124244 _runtime‚5057.345512035 0loss/FRt2/assignment_loss‚1.2409508135169744 1loss/SRqqt2/detection_loss‚0.8015026834738606 /loss/FBtדá+2/detection_loss‚0.4732176980956696 0loss/FRt3/assignment_loss‚3.5497247061946173 /loss/FRt2/detection_loss‚1.5104848337359726 .loss/FRt4/detection_loss‚2.461669290211143 trainer/global_step‚5249 % _timestamp‚1.7884215865956967e+09 0loss/FRt1/assignment_loss‚3.8963488892811102 1loss/SRqqt2/assignment_loss‚0.987968820380047 1loss/SRqqt4/detection_loss‚1.8887803457221206 /loss/FBt1/detection_loss‚2.6617568797535367 /loss/FBt4/detection_loss‚2.7097835540771484 2loss/SRqqt3/assignment_loss‚1.9877961220155502Úf¿Ša¦Û %loss/total_loss‚44.45386027124244 _runtime‚5057.345512035 0loss/FRt2/assignment_loss‚1.2409508135169744 1loss/SRqqt2/detection_loss‚0.8015026834738606 /loss/FBt2/detection_loss‚0.4732176980956696 0loss/FRt3/assignment_loss‚3.5497247061946173 /loss/FRt2/detection_loss‚1.5104848337359726 .loss/FRt4/detection_loss‚2.461669290211143 trainer/global_step‚5249 % _timestamp‚1.7884215865956967e+09 0loss/FRt1/assignment_loss‚3.8963488892811102 1loss/SRqqt2/assignment_loss‚0.987968820380047 1loss/SRqqt4/detection_loss‚1.8887803457221206 /loss/FBt1/detection_loss‚2.6617568797535367 /loss/FBt4/detection_loss‚2.7097835540771484 2loss/SRqqt3/assignment_loss‚1.9877961220155502 1loss/SRqqt3/detection_loss‚1.9230529049347187 .loss/FBt3/detection_loss‚3.974043369293213 _step‚431 0loss/FRt4/assignment_loss‚3.2491766621426836 2loss/SRqqt1/assignment_loss‚1.6438105429212253 0loss/FBt1/assignment_loss‚0.6991546335112717 /loss/FRt1/detection_loss‚2.3143215455660004 /loss/FRt3/detection_loss‚2.5566875880414788 epoch‚4 1loss/SRqqt4/assignment_loss‚1.456762635814292 1loss/FBt2/assignment_loss‚0.17165244916967026 0loss/FBt3/assignment_loss‚0.1323621767029787 1loss/FBt4/assignment_loss‚0.07428035450478394 0loss/SRqqt1/detection_loss‚2.088570772182374¯œx-b§‚(:… ßËäÔÿÄ›x gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.392' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`6y¨‚(:î ßËäÔÞ‘·{ disk.rbd0.in‚0 disk.sda3.out‚ 280.83984375 disk.sda1.in‚ 0.05078125$ disk.nvme0n1.out‚22565.677734375! disk.dm-0.out‚22565.677734375( proc.memory.availableMB‚ 336.30078125" proc.memory.rssMB‚ 2862.7578125 disk.rbd1.out‚ 2.51611328125& disk.nvme0c0n1.out‚22565.677734375 disk.sda1.out‚0 disk.sda3.in‚7.3125$ memory_percent‚97.94738292694092) proc.memory.percent‚17.47288703918457 network.sent‚ 1.2289736e+07 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd2.out‚ 170.669921875 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.nvme0n1.in‚ 4.9765625 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 280.83984375 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚ 4.9765625 disk.sda.in‚ 7.36328125 disk.sda.out‚ 280.87890625 cpu‚25.284565935856374 network.recv‚ 1.800528e+06 disk.dm-0.in‚ 4.9765625 disk.rbd0.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚0) disk./.usagePercent‚73.71586672026577% disk./.usageGB‚1098.4196853637695ãjÄC©‚(:„ îËäÔãÎœx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.972' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5944 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ~Ñfª‚(:Û îËäÔú€†| disk.sda1.in‚ 0.05078125 network.sent‚ 1.239841e+07% disk./.usageGB‚1098.4196853637695$ memory_percent‚97.94766902923584( proc.memory.availableMB‚ 336.25390625" proc.memory.rssMB‚ 2862.7578125) proc.memory.percent‚17.47288703918457 proc.cpu.threads‚17# disk.nvme0n1.out‚22566.65234375 disk.rbd2.out‚ 170.748046875 disk.rbd0.in‚0 disk.drbd1777.out‚0% disk.nvme0c0n1.out‚22566.65234375 disk.dm-1.in‚7.3125 disk.dm-1.out‚281.5 disk.rbd3.in‚0 disk.rbd3.out‚0 network.recv‚ 1.806186e+06 disk.sda.in‚ 7.36328125 disk.sda.out‚ 281.5390625 disk.dm-0.out‚22566.65234375 disk.sda2.out‚ 0.0390625 disk.sda1.out‚0) disk./.usagePercent‚73.71586672026577 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 cpu‚25.283920102591885 disk.dm-0.in‚ 4.9765625 disk.nvme0c0n1.in‚ 4.9765625 disk.drbd1777.in‚ 1.03515625 disk.sda3.in‚7.3125 disk.sda3.out‚281.5 disk.rbd1.in‚0 disk.rbd1.out‚ 2.51611328125 disk.nvme0n1.in‚ 4.9765625 disk.sda2.in‚0Æ/5Ž«‚(:ƒ ýËäÔü•®x gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚99.185' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.837 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûÏ2i¬‚(:Þ ýËäÔ³”‘| disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd4.in‚0! disk.nvme0c0n1.out‚ 22665.6875 disk.rbd1.out‚ 2.51611328125 disk.nvme0n1.in‚ 4.9765625 disk.dm-1.out‚ 282.05078125" proc.memory.rssMB‚ 2862.7578125 network.sent‚ 1.240482e+07 disk.sda1.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 4.9765625 disk.rbd0.out‚0( proc.memory.availableMB‚ 336.25390625 network.recv‚ 1.810995e+06% disk./.usageGB‚1098.4196853637695 disk.sda2.in‚0 disk.dm-1.in‚7.3125$ memory_percent‚97.94766902923584 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.nvme0n1.out‚ 22665.6875 disk.dm-0.out‚ 22665.6875 disk.rbd2.in‚0 disk.drbd1777.out‚0) proc.memory.percent‚17.47288703918457 disk.rbd4.out‚0 disk.rbd1.in‚0 cpu‚25.283234819358423 disk.sda3.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚ 282.08984375 disk.sda3.out‚ 282.05078125 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 172.267578125) disk./.usagePercent‚73.71586672026577 disk.rbd3.in‚0°ÆË.2­- _wandbruntime‚5102  _runtime‚5102 AâHÇ®Á & lr-AdamW/pg1‚0.0005536458304863337 & lr-AdamW/pg2‚0.0005536458304863337 trainer/global_step‚5299 % _timestamp‚1.7884216317608647e+09 _runtime‚5102.510833107 _step‚432 e–´Ì¯Æ % _timestamp‚1.7884216317608647e+09 _runtime‚5102.510833107 _step‚432 & lr-AdamW/pg1‚0.0005536458304863337 & lr-AdamW/pg2‚0.0005536458304863337 trainer/global_step‚5299°Oî8È2°- _wandbruntime‚5102  _runtime‚5102­}«á'±ÛO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97686727 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9839482259456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9810698329656 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235569166 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137882439 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235543494 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97686725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57151700 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9810921978804 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚225122460895232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 _runtime‚5102.511868528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚219635129874432 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137856769 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚219634883667968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35613951 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚209795647614976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9839482259456 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57151698 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209795629612032 _step‚433 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9839254055936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚225122484835328 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚208464912306100 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚219634883667968 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚208464671617592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137856769 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92765649 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 % _timestamp‚1.7884216317616405e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97686727 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0iáøÀæ'²àO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137882439 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚219634883667968 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚208464671617592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137856769 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92765649 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 % _timestamp‚1.7884216317616405e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97686727 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97686727 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9839482259456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9810698329656 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235569166 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137882439 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235543494 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97686725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57151700 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9810921978804 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚225122460895232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 _runtime‚5102.511868528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚219635129874432 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137856769 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚219634883667968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35613951 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚209795647614976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9839482259456 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57151698 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209795629612032 _step‚433 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9839254055936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚225122484835328 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚208464912306100 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235586764 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 trainer/global_step‚5299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚225163654531584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 291876864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97704446 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235178635662336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚198690292785152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35621208 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97704444 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 315816960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 E?DeviceStatsMonitor.on_train_baê‰çâÌtch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚5103.399211456 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9812530729256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235586764 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97704446 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92782605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235178319845376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137882320 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57162053 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚208502806475048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9841319967232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235612462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9841091750400 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209833967556096 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137908016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92783263 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚434 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚219675305526272 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137908016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92783263 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚434 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚219675305526272 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235586764 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 trainer/global_step‚5299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚225163654531584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 291876864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97704446 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235178635662336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚198690292785152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35621208 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97704444 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 315816960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚5103.399211456 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9812530729256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235586764 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97704446 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92782605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235178319845376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137882320 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57162053 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚208502806475048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9841319967232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235612462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9841091750400 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209833967556096²w jO2¶- _wandbruntime‚5103  _runtime‚5103R—zeX·Ò 2loss/SRqqt2/assignment_loss‚1.0020857608398877 0loss/FBt1/assignment_loss‚0.5918263251977888 _runtime‚5103.400335744 /loss/FRt1/assignment_loss‚4.783078627708631 .loss/FBt1/detection_loss‚2.267991065979004 epoch‚4 0loss/FRt2/assignment_loss‚2.0456922839085263 2loss/SRqqt3/assignment_loss‚1.5280243443215595 1loss/FBt3/assignment_loss‚0.24754731683060527 1loss/SRqqt3/detection_loss‚1.8833304088489682 0loss/SRqqt4/detection_loss‚2.015035999417305 %loss/total_loss‚45.47673814885395 1loss/SRqqt1/assignment_loss‚1.990012461585658 /loss/FRt1/detection_loss‚2.6504681622370696 /loss/FBt3/detection_loss‚2.3892392963171005 $ _timestamp‚1.788421632650376e+09 0loss/FRt3/assignment_loss‚2.4228048889260543 1loss/SRqqt4/assignment_loss‚1.701348389685154 .loss/FRt3/detection_loss‚2.815706639697677 trainer/global_step‚5299 1loss/FBt4/assignment_loss‚0.15626067526838078 /loss/FBt2/detection_loss‚0.5555578324433771 .loss/FBt4/detection_loss‚2.681898662022182 _step‚435 0loss/FRt4/assignment_loss‚3.6623100638389587 1loss/FBt2/assignment_loss‚0.16253708287168805 /loss/FRt2/detection_loss‚1.5235763408243657 /loss/FRt4/detection_loss‚2.5781977411472434 /loss/SRqqt1/detection_loss‚2.98737781388419 1loss/SRqqt2/detection_loss‚0.8348299650525725Zÿû]¸× 2loss/SRqqt2/assignment_loss‚1.0020857608398877 0loss/FBt1/assignment_loss‚0.5918263251977888 _runtime‚5103.400335744 /loss/FRt1/assignment_loss‚4.783078627708631 .loss/FBt1/detection_loss‚2.267991065979004 epoch‚4 0loss/FRt2/assignment_loss‚2.0456922839085263 2loss/SRqqt3/assignment_loss‚1.5280243443215595 1loss/FBt3/assignment_loss‚0.24754731683060527 1loss/SRqqt3/detection_loss‚1.8833304088489682 0loss/SRqqt4/detection_loss‚2.015035999417305 %loss/total_loss‚45.47673814885395 1loss/SRqqt1/assignment_loss‚1.990012461585658 /loss/FRt1/detection_loss‚2.6504681622370696 /loss/FBt3/detection_loss‚2.3892392963171005 $ _timestamp‚1.788421632650376e+09 0loss/FRt3/assignment_loss‚2.4228048889260543 1loss/SRqqt4/assignment_loss‚1.701348389685154 .loss/FRt3/detection_loss‚2.815706639697677 trainer/global_step‚5299 1loss/FBt4/assignment_loss‚0.15626067526838078 /loss/FBt2/detection_loss‚0.5555578324433771 .loss/FBt4/detection_loss‚2.681898662022182 _step‚435 0loss/FRt4/assignment_loss‚3.6623100638389587 1loss/FBt2/assignment_loss‚0.16253708287168805 /loss/FRt2/detection_loss‚1.5235763408243657 /loss/FRt4/detection_loss‚2.5781977411472434 /loss/SRqqt1/detection_loss‚2.98737781388419 1loss/SRqqt2/detection_loss‚0.8348299650525725³¾,|›¹‚(: ŒÌäÔ­˜¢x gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žë/vº‚(:ë ŒÌäÔͦËz disk.sda3.in‚7.3125 disk.rbd1.out‚ 2.51611328125 disk.rbd2.out‚ 172.404296875 disk.sda3.out‚ 282.48828125' proc.memory.availableMB‚ 336.7109375" proc.memory.rssMB‚ 2862.7578125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 282.48828125) proc.memory.percent‚17.47288703918457 network.sent‚ 1.2409098e+07 network.recv‚ 1.813697e+06 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚ 4.9765625 cpu‚25.28252143787163 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125$ memory_percent‚97.94487953186035% disk./.usageGB‚1098.4196891784668 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0$ disk.nvme0n1.out‚22765.162109375 disk.rbd2.in‚0( disk./.usagePercent‚73.7158669762733& disk.nvme0c0n1.out‚22765.162109375 disk.rbd3.out‚0 disk.dm-0.in‚ 4.9765625! disk.dm-0.out‚22765.162109375 disk.sda2.in‚0 disk.rbd4.in‚0 disk.sda.out‚ 282.52734375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.out‚0y’/»‚(:… ›ÌäÔžüÕy gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âPo¼‚(:ä ›ÌäÔÿÂ{ network.recv‚ 1.819877e+06 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9765625 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda3.out‚ 282.91796875 disk.dm-0.out‚ 22864.265625 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0( proc.memory.availableMB‚ 336.70703125 cpu‚25.281835539607563 disk.rbd2.out‚ 172.404296875# disk.nvme0c0n1.out‚ 22864.265625 disk.rbd3.in‚0" proc.memory.rssMB‚ 2862.7578125 proc.cpu.threads‚17 disk.sda.out‚ 282.95703125! disk.nvme0n1.out‚ 22864.265625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.9765625( disk./.usagePercent‚73.7158669762733 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.rbd1.out‚ 2.51611328125 disk.dm-1.out‚ 282.91796875 disk.rbd2.in‚0 disk.drbd1777.out‚0$ memory_percent‚97.94490337371826 disk.dm-0.in‚ 4.9765625 disk.sda2.out‚ 0.0390625 disk.rbd4.out‚0 disk.rbd3.out‚0) proc.memory.percent‚17.47288703918457 network.sent‚ 1.2517766e+07% disk./.usageGB‚1098.4196891784668 disk.dm-1.in‚7.3125 disk.sda1.in‚ 0.05078125éÉ¡½‚(:„ ªÌäԇѕx gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.045' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.809 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥Qìãq¾‚(:æ ªÌäÔ„·ó{ network.recv‚ 1.824239e+06% disk./.usageGB‚1098.4195022583008 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚283.625" proc.memory.rssMB‚ 2862.7578125 disk.dm-1.in‚7.3125 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd1.in‚0) disk./.usagePercent‚73.71585443190412 disk.sda1.out‚0) proc.memory.percent‚17.47288703918457 cpu‚25.281207647739688! disk.rbd2.out‚172.52197265625 disk.drbd1777.out‚0 disk.sda3.out‚ 283.5859375 disk.dm-0.out‚22865.12890625 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125$ memory_percent‚97.94785976409912 disk.rbd4.out‚0 network.sent‚ 1.2523642e+07 disk.rbd2.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 336.22265625% disk.nvme0c0n1.out‚22865.12890625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 4.9765625# disk.nvme0n1.out‚22865.12890625 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 283.5859375 disk.rbd0.out‚0 disk.rbd1.out‚ 2.51611328125 disk.dm-0.in‚ 4.9765625 disk.sda2.out‚ 0.03906255Êà²2¿- _wandbruntime‚5148  _runtime‚5148b|¨&ÆÀÀ & lr-AdamW/pg1‚0.0005482736256831622 & lr-AdamW/pg2‚0.0005482736256831622 trainer/global_step‚5349 $ _timestamp‚1.788421677951736e+09 _runtime‚5148.701677695 _step‚436éhtËÁÅ _runtime‚5148.701677695 _step‚436 & lr-AdamW/pg1‚0.0005482736256831622 & lr-AdamW/pg2‚0.0005482736256831622 trainer/global_step‚5349 $ _timestamp‚1.788421677951736e+09´F'q±2Â- _wandbruntime‚5148  _runtime‚5148HíW)è&ÃÚO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚200469096180736 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210371428081600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 237736018 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚237287166797312 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9902331900864 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 237710346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211712544818176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9902555550012 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211712526815232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98572677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221643680008192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9931153192960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139163341 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚5148.703329899 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚221643680008192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227180945708032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _step‚437 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚221643926214656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211712526815232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93610042 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚221643926214656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚5349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98572675 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35941256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139137671 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98572675 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35940594 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚200469113220096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211712544818176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9931381396480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.sœ¼ßømall_pool.current‚ 223649148 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚01F¤å'ÄßO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚237287166797312 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9902331900864 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 237710346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211712544818176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9902555550012 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211712526815232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98572677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221643680008192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9931153192960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139163341 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚5148.703329899 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚221643680008192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227180945708032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _step‚437 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚221643926214656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211712526815232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93610042 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚221643926214656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚5349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98572675 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35941256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139137671 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98572675 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35940594 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚200469113220096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211712544818176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9931381396480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚200469096180736 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210371428081600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 237736018 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800µmí{¹2Å- _wandbruntime‚5149  _runtime‚5149æ­ë&ÆåM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221684101866496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚211750864759296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 237753616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10108092068864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9932990887424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 % _timestamp‚1.7884216788116422e+09 _step‚438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚221684101866496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237779314 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57679805 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35947447 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9932990887424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚221683855646720 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200505398638592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 237753616 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚237330498124288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210409803627772 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9933219104256 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98590396 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98590396 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚211750882762240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚227222115404288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9904387949820 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98590394 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93626614 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚237330207473152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 237779314 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57679803 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93627252 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10108358779904 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98590394 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139188918 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139163222 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9904164300464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 trainer/global_step‚5349 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200505415677952 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5149.562598881?®¤ð&ÇêM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200505398638592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 237753616 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚237330498124288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210409803627772 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9933219104256 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98590396 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98590396 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚211750882762240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚227222115404288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9904387949820 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98590394 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93626614 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚237330207473152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 237779314 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57679803 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93627252 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10108358779904 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98590394 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139188918 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139163222 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9904164300464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 trainer/global_step‚5349 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200505415677952 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5149.562598881 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221684101866496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚211750864759296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 237753616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10108092068864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9932990887424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 % _timestamp‚1.7884216788116422e+09 _step‚438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚221684101866496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237779314 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57679805 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35947447 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9932990887424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚221683855646720 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992¶ï7Ç2È- _wandbruntime‚5149  _runtime‚5149nÄvXÊÓÄ & lr-AdamW/pg1‚0.0005428707624824227 & lr-AdamW/pg2‚0.0005428707624824227 trainer/global_step‚5399 % _timestamp‚1.7884217237826638e+09 _runtime‚ 5194.5325682 _step‚440¸ ”{2Ô- _wandbruntime‚5194  _runtime‚5194ȬgPå'ÕßO `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10199739180544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202284219073536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140444243 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94455055 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚223652722554880 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99458627 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140444243 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213629442021376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239877198 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9994189121220 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10023052329984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 _step‚441 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚213629424018432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212278184545608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94455737 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229239430520832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58187198 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239439169701376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140418573 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚223652722554880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140418573 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚223652476348416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36267857 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213629424018432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58187200 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223652476348416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99458625 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚229239454460928 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 _runtime‚5194.533481861 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 239902870 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99458627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213629442021376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10200012196352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212278425234116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚239439466657280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202284236112896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10023280533504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0¡XPâê'ÖäO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229239430520832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58187198 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239439169701376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140418573 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚223652722554880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140418573 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚223652476348416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36267857 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213629424018432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58187200 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223652476348416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99458625 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚229239454460928 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 _runtime‚5194.533481861 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 239902870 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99458627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213629442021376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10200012196352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212278425234116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚239439466657280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202284236112896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10023280533504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10199739180544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202284219073536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140444243 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94455055 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚223652722554880 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99458627 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140444243 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213629442021376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239877198 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9994189121220 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10023052329984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 _step‚441 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚213629424018432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212278184545608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94455737 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824¹©èú#2×- _wandbruntime‚5195  _runtime‚5195P±©TiØéM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 239920468 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10201878335488 _step‚442 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140444124 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94472562 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 % _timestamp‚1.7884217246635373e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140469820 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213667779965440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10025118241280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36275007 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10024890024448 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚5195.414527865 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚223692898206720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239482211841536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99476346 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229280600217088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94471918 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99476344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213667779965440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239946166 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚213667761962496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213667761962496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9996021521028 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 239920468 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58197555 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202320538570752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99476346 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223692651986944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140444124 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212316319403064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 trainer/global_step‚5399 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202320521531392 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239946166 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10201611624448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561¦ S­ô&ÙîM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140469820 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213667779965440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10025118241280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36275007 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10024890024448 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚5195.414527865 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚223692898206720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239482211841536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99476346 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229280600217088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94471918 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99476344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213667779965440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239946166 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚213667761962496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213667761962496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9996021521028 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 239920468 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58197555 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202320538570752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99476346 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223692651986944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140444124 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212316319403064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 trainer/global_step‚5399 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202320521531392 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239946166 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10201611624448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 239920468 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10201878335488 _step‚442 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140444124 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94472562 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 % _timestamp‚1.7884217246635373e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896º+¶]2Ú- _wandbruntime‚5195  _runtime‚5195ª‚§CRÛÌ 0loss/FRt3/assignment_loss‚3.1625322967767717 0loss/FBt1/assignment_loss‚0.4862944283099337 0loss/FBt2/assignment_loss‚0.1423559135200916 0loss/SRqqt2/detection_loss‚1.058864000889044 2loss/SRqqt1/assignment_loss‚1.7028826167568183 1loss/SRqqt2/assignment_loss‚1.061868979776692 /loss/FRt2/detection_loss‚1.5773946029959984 .loss/FRt4/detection_loss‚2.513277399998458 % _timestamp‚1.7884217246657827e+09 1loss/FBt4/assignment_loss‚0.17704662905033627 0loss/FRt2/assignment_loss‚1.6152954000347066 /loss/FRt4/assignment_loss‚4.595550570732508 0loss/FBt3/assignment_loss‚0.3654071889921195 1loss/SRqqt3/detection_loss‚1.9436714214086532 0loss/SRqqt4/detection_loss‚2.042195688933134 _runtime‚ 5195.41560813 1loss/SRqqt3/assignment_loss‚1.729098022133112 .loss/FRt3/detection_loss‚2.569059120118618 %loss/total_loss‚45.41348748127888 _step‚443 1loss/SRqqt4/assignment_loss‚1.652714246418327 .loss/FRt1/detection_loss‚2.610116096585989 /loss/FBt1/detection_loss‚2.4416851469061593 /loss/FBt2/detection_loss‚0.5764839937668536 -loss/FBt3/detection_loss‚2.37685387995508 epoch‚4 trainer/global_step‚5399 0loss/SRqqt1/detection_loss‚2.383207096503331 /loss/FBt4/detection_loss‚2.5324188272158303 /loss/FRt1/assignment_loss‚4.097213913500309|WÜÑ /loss/FRt2/detection_loss‚1.5773946029959984 .loss/FRt4/detection_loss‚2.513277399998458 % _timestamp‚1.7884217246657827e+09 1loss/FBt4/assignment_loss‚0.17704662905033627 0loss/FRt2/assignment_loss‚1.6152954000347066 /loss/FRt4/assignment_loss‚4.595550570732508 0loss/FBt3/assignment_loss‚0.3654071889921195 1loss/SRqqt3/detection_loss‚1.9436714214086532 0loss/SRqqt4/detection_loss‚2.042195688933134 _runtime‚ 5195.41560813 1loss/SRqqt3/assignment_loss‚1.729098022133112 .loss/FRt3/detection_loss‚2.569059120118618 %loss/total_loss‚45.41348748127888 _step‚443 1loss/SRqqt4/assignment_loss‚1.652714246418327 .loss/FRt1/detection_loss‚2.610116096585989 /loss/FBt1/detection_loss‚2.4416851469061593 /loss/FBt2/detection_loss‚0.5764839937668536 -loss/FBt3/detection_loss‚2.37685387995508 epoch‚4 trainer/global_step‚5399 0loss/SRqqt1/detection_loss‚2.383207096503331 /loss/FBt4/detection_loss‚2.5324188272158303 /loss/FRt1/assignment_loss‚4.097213913500309 0loss/FRt3/assignment_loss‚3.1625322967767717 0loss/FBt1/assignment_loss‚0.4862944283099337 0loss/FBt2/assignment_loss‚0.1423559135200916 0loss/SRqqt2/detection_loss‚1.058864000889044 2loss/SRqqt1/assignment_loss‚1.7028826167568183 1loss/SRqqt2/assignment_loss‚1.061868979776692»<õìšÝ‚(: æÌäÔ÷â²x gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.579' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.915799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vÃk)yÞ‚(:î æÌäÔŸÌ·{ disk.rbd4.in‚0 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 2.51611328125 disk.nvme0n1.in‚ 4.9765625 cpu‚25.27841716239767 proc.cpu.threads‚17 network.sent‚ 1.2649874e+07 disk.dm-1.in‚7.3125! disk.rbd2.out‚174.25244140625( disk.nvme0c0n1.out‚23163.25244140625 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 285.71875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2862.7578125 disk.rbd3.in‚0 disk.sda3.in‚7.3125$ memory_percent‚97.94909954071045 disk.rbd4.out‚0( proc.memory.availableMB‚ 336.01953125 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.dm-0.in‚ 4.9765625# disk.dm-0.out‚23163.25244140625 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 285.71875 disk.rbd2.in‚0 disk.sda.out‚ 285.7578125 disk.rbd1.in‚0) disk./.usagePercent‚73.71586953634865% disk./.usageGB‚1098.4197273254395 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd3.out‚0& disk.nvme0n1.out‚23163.25244140625) proc.memory.percent‚17.47288703918457 network.recv‚ 1.842947e+06 disk.rbd0.in‚0¿ŒÏß‚(:… õÌäÔìì­x gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s¡ îyà‚(:î õÌäÔ“øá{" disk.dm-0.out‚23262.3310546875$ memory_percent‚97.94917106628418 proc.cpu.threads‚17! disk.rbd2.out‚174.25244140625 network.sent‚ 1.2759029e+07 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda2.in‚0 disk.drbd1777.out‚0' disk./.usagePercent‚73.715875936537 disk.rbd4.out‚0' disk.nvme0c0n1.out‚23262.3310546875 disk.dm-1.out‚ 286.17578125$ disk./.usageGB‚1098.419822692871 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.out‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2862.7578125 network.recv‚ 1.849322e+06 disk.sda.out‚ 286.21484375 disk.rbd1.in‚0 disk.rbd1.out‚ 2.51611328125 disk.nvme0n1.in‚ 4.9765625' proc.memory.availableMB‚ 336.0078125) proc.memory.percent‚17.47288703918457 disk.rbd3.out‚0 disk.sda3.in‚7.3125% disk.nvme0n1.out‚23262.3310546875 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 286.17578125 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 cpu‚25.27780849693847 disk.dm-0.in‚ 4.9765625 disk.dm-1.in‚7.3125¬HiÒá‚(:… „ÍäÔ¢ŽÇx gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[m÷´vâ‚(:ë „ÍäÔÈÕ÷z disk.dm-0.in‚ 4.9765625 disk.sda.in‚ 7.36328125 cpu‚25.277167269544076# disk.dm-0.out‚23361.66162109375 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0( proc.memory.availableMB‚ 335.76953125 disk.rbd1.in‚0 disk.dm-1.in‚7.3125 disk.sda3.out‚ 286.703125# memory_percent‚97.9506254196167 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 1.2764252e+07 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.71587977665001 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9765625( disk.nvme0c0n1.out‚23361.66162109375 disk.nvme0n1.in‚ 4.9765625 disk.rbd2.in‚0 disk.sda.out‚ 286.7421875 network.recv‚ 1.8528e+06) proc.memory.percent‚17.47288703918457 disk.rbd1.out‚ 2.51611328125& disk.nvme0n1.out‚23361.66162109375 disk.dm-1.out‚ 286.703125 disk.rbd2.out‚174.3544921875 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.sda3.in‚7.3125" proc.memory.rssMB‚ 2862.7578125 disk.sda1.in‚ 0.05078125# disk./.usageGB‚1098.41987991333wû|2ã- _wandbruntime‚5240  _runtime‚5240¸ ¢ÇäÁ & lr-AdamW/pg1‚0.0005374383668327081 & lr-AdamW/pg2‚0.0005374383668327081 trainer/global_step‚5449 % _timestamp‚1.7884217701592221e+09 _runtime‚5240.909158504 _step‚444ZUòBÌåÆ _runtime‚5240.909158504 _step‚444 & lr-AdamW/pg1‚0.0005374383668327081 & lr-AdamW/pg2‚0.0005374383668327081 trainer/global_step‚5449 % _timestamp‚1.7884217701592221e+09¼ëb²£2æ- _wandbruntime‚5240  _runtime‚5240g±4ë'çåO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95299957 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36595007 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚445 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215546321221632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58704948 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚204099359005696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10085822692428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚225661518895104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241591300796416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚214184941009616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚225661272688640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚225661518895104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10115179670528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10293361522688 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141699475 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141725145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10293088506880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10114951467008 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10085599043280 trainer/global_step‚5449 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231297915333632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225661272688640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141725145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215546339224576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95299304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 % _timestamp‚1.7884217701599653e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141699475 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58704950 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36594356 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚231297939273728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10115179670528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242069722 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225661272688640 UG°lÀ#DeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141725145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215546339224576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95299304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 % _timestamp‚1.7884217701599653e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141699475 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58704950 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36594356 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚231297939273728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10115179670528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242069722 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36595007 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚445 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215546321221632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58704948 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚204099359005696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10085822692428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚225661518895104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241591300796416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚214184941009616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚225661272688640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚225661518895104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10115179670528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10293361522688 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141699475 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141725145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10293088506880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10114951467008 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10085599043280 trainer/global_step‚5449 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231297915333632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884217710553076e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚231339108969984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10294946422784 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141725026 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214223316555788 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚204135644424192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 _runtime‚5241.805970944 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95316206 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141750722 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215584659165696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215584677168640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100362294 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100362294 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141725026 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 283488256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10087655092236 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 trainer/global_step‚5449 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95316871 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10295229911040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141750722 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242113018 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100362296 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242087320 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10117017378304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231339085029888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10117017378304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242113018 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36600903 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 307428352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58715305 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214223075867072iZÄIú&ëôM EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95316206 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141750722 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215584659165696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215584677168640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100362294 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100362294 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141725026 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 283488256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10087655092236 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 trainer/global_step‚5449 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95316871 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10295229911040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141750722 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242113018 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100362296 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242087320 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10117017378304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231339085029888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10117017378304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242113018 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36600903 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 307428352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58715305 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214223075867072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884217710553076e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚231339108969984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10294946422784 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141725026 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214223316555788 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚204135644424192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 _runtime‚5241.805970944¾B\ôÕ2ì- _wandbruntime‚5241  _runtime‚52419e;VíÐ 1loss/SRqqt2/assignment_loss‚0.891192609713854 epoch‚4 trainer/global_step‚5449 /loss/FRt2/detection_loss‚1.6419130032882094 /loss/FRt1/assignment_loss‚4.113558265897963 0loss/SRqqt3/detection_loss‚1.835019591179761 &loss/total_loss‚43.011772497137386 0loss/FRt2/assignment_loss‚1.1818877905607224 .loss/FRt1/detection_loss‚2.377116041878859 .loss/FRt3/detection_loss‚2.518964354808514 _runtime‚5241.806917848 1loss/SRqqt1/assignment_loss‚2.070837664604187 1loss/SRqqt4/detection_loss‚2.1464431080080213 .loss/FBt1/detection_loss‚2.470582979917526 /loss/FBt2/detection_loss‚0.6525206724868009 /loss/FBt4/detection_loss‚2.8491592407226562 _step‚447 1loss/SRqqt4/assignment_loss‚1.406839972406271 1loss/FBt1/assignment_loss‚0.05967374742031097 0loss/SRqqt1/detection_loss‚2.084684383124113 0loss/FRt3/assignment_loss‚2.8888336572891626 0loss/FRt4/assignment_loss‚3.6456788500150044 /loss/FBt3/assignment_loss‚0.145080187326433 0loss/FBt4/assignment_loss‚0.1995112869578103 .loss/FRt4/detection_loss‚2.494624956448873 .loss/FBt3/detection_loss‚2.447237859169642 % _timestamp‚1.7884217710569847e+09 2loss/SRqqt3/assignment_loss‚1.8534936956383965 1loss/FBt2/assignment_loss‚0.13007139829259992 1loss/SRqqt2/detection_loss‚0.9068471799817002câîa[îÕ 0loss/FRt2/assignment_loss‚1.1818877905607224 .loss/FRt1/detection_loss‚2.377116041878859 .loss/FRt3/detection_loss‚2.518964354808514 _runtime‚5241.806917848 1loss/SRqqt1/assignment_loss‚2.070837664604187 1loss/SRqqt4/detection_loss‚2.1464431080080213 .loss/FBt1/detection_loss‚2.470582979917526 /loss/FBt2/detection_loss‚0.6525206724868009 /loss/FBt4/detection_loss‚2.8491592407226562 _step‚447 1loss/SRqqt4/assignment_loss‚1.406839972406271 1loss/FBt1/assignment_loss‚0.05967374742031097 0loss/SRqqt1/detection_loss‚2.084684383124113 0loss/FRt3/assignment_loss‚2.8888336572891626 0loss/FRt4/assignment_loss‚3.6456788500150044 /loss/FBt3/assignment_loss‚0.145080187326433 0loss/FBt4/assignment_loss‚0.1995112869578103 .loss/FRt4/detection_loss‚2.494624956448873 .loss/FBt3/detection_loss‚2.447237859169642 % _timestamp‚1.7884217710569847e+09 2loss/SRqqt3/assignment_loss‚1.8534936956383965 1loss/FBt2/assignment_loss‚0.13007139829259992 1loss/SRqqt2/detection_loss‚0.9068471799817002 1loss/SRqqt2/assignment_loss‚0.891192609713854 epoch‚4 trainer/global_step‚5449 /loss/FRt2/detection_loss‚1.6419130032882094 /loss/FRt1/assignment_loss‚4.113558265897963 0loss/SRqqt3/detection_loss‚1.835019591179761 &loss/total_loss‚43.011772497137386¿eU¿†Žï‚(:ƒ “ÍäÔÌ߬x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àNëoð‚(:ô “ÍäÔéðË{ disk.drbd1777.in‚ 1.03515625 disk.sda2.out‚ 0.0390625 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 287.19140625 disk.sda3.out‚ 287.15234375 disk.dm-1.out‚ 287.15234375 disk.rbd3.in‚0( disk.nvme0c0n1.out‚23460.72412109375 disk.sda1.in‚ 0.05078125$ memory_percent‚97.94924259185791( proc\8N›.memory.availableMB‚ 335.99609375# disk.dm-0.out‚23460.72412109375 disk.rbd2.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.47288703918457 disk.dm-1.in‚7.3125 disk.rbd2.out‚174.3544921875 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.nvme0n1.in‚ 4.9765625 disk.sda2.in‚0 cpu‚25.27646762780211 network.recv‚ 1.857286e+06% disk./.usageGB‚1098.4199295043945 disk.rbd4.in‚0" proc.memory.rssMB‚ 2862.7578125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.dm-0.in‚ 4.9765625) disk./.usagePercent‚73.71588310474796 disk.sda3.in‚7.3125 disk.rbd1.out‚ 2.51611328125& disk.nvme0n1.out‚23460.72412109375 network.sent‚ 1.2770377e+07 disk.rbd1.in‚0,®Kñ‚(:… ¢ÍäÔ‚‘¨x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oížî}ò‚(:ò ¢ÍäÔå£Åz disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 4.9765625 disk.dm-1.in‚7.3125 disk.sda1.out‚0& disk.nvme0n1.out‚23560.79931640625 proc.cpu.threads‚17% disk./.usageGB‚1098.4201126098633 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.9765625( disk.nvme0c0n1.out‚23560.79931640625 disk.rbd1.out‚ 2.51611328125" proc.memory.rssMB‚ 2862.7578125) proc.memory.percent‚17.47288703918457 cpu‚25.275883254053124# disk.dm-0.out‚23560.79931640625 disk.rbd4.in‚0$ memory_percent‚97.94950485229492 network.sent‚ 1.2878808e+07 disk.dm-1.out‚ 287.67578125 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125& proc.memory.availableMB‚ 335.953125 network.recv‚ 1.863242e+06 disk.rbd2.out‚176.0068359375 disk.sda.out‚ 287.71484375 disk.nvme0n1.in‚ 4.9765625 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda3.out‚ 287.67578125( disk./.usagePercent‚73.7158953931096 disk.drbd1777.in‚ 1.03515625œ&4†›ó‚(: ±ÍäÔõˆ¾x gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.872' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.974400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UwÖ‡sô‚(:è ±ÍäÔú¶Ç{ cpu‚25.27523474303034 proc.cpu.threads‚17 disk.rbd1.in‚0# disk.dm-0.out‚23561.05712890625 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚176.0068359375 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0( proc.memory.availableMB‚ 335.44921875 disk.rbd1.out‚ 2.51611328125 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda3.out‚288.125 disk.nvme0n1.in‚ 4.9765625& disk.nvme0n1.out‚23561.05712890625 network.recv‚ 1.868048e+06 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 network.sent‚ 1.2885112e+07% disk./.usageGB‚1098.4201126098633 disk.sda3.in‚7.3125 disk.dm-0.in‚ 4.9765625 disk.nvme0c0n1.in‚ 4.9765625 disk.sda.in‚ 7.36328125 disk.sda.out‚ 288.1640625" proc.memory.rssMB‚ 2862.7578125( disk./.usagePercent‚73.7158953931096 disk.dm-1.in‚7.3125 disk.rbd4.out‚0 disk.rbd3.out‚0$ memory_percent‚97.95258045196533) proc.memory.percent‚17.47288703918457 disk.dm-1.out‚288.125 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚23561.05712890625¸ •2õ- _wandbruntime‚5287  _runtime‚5287¾q¢ÇöÁ _step‚448 & lr-AdamW/pg1‚0.0005319775708371295 & lr-AdamW/pg2‚0.0005319775708371295 trainer/global_step‚5499 % _timestamp‚1.7884218166680274e+09 _runtime‚5287.417965106… XÌ÷Æ & lr-AdamW/pg1‚0.0005319775708371295 & lr-AdamW/pg2‚0.0005319775708371295 trainer/global_step‚5499 % _timestamp‚1.7884218166680274e+09 _runtime‚5287.417965106 _step‚448À‘LFë2ø- _wandbruntime‚5287  _runtime‚5287ÜJ4ªë'ùåO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244210902 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244236574 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142980377 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚205914481898496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143006047 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59222700 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚243742814433792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10206850604032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚205914464859136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚243743103001088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227670069028864 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59222698 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚216091938162132 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142980377 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚233356400146432 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227670069028864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 264627200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10177456263636 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96140891 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101230527 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10207078807552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227670315235328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101230525 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚217463218424832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10386414287360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10206850604032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96141539 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 288567296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142980377 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚205914481898496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143006047 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59222700 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚243742814433792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10206850604032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚205914464859136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚243743103001088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227670069028864 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59222698 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚216091938162132 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142980377 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚233356400146432 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227670069028864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 264627200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10177456263636 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96140891ÁUVÿM2û- _wandbruntime‚5288  _runtime‚5288îå vüïM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚205950767316992 trainer/global_step‚5499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244254172 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227710490887168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚243786145189376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚646 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216129832331080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚227710490887168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚5288.318205131 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101248244 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96157870 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚233397593782784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚233397569842688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227710244667392 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101248244 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚450 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚205950784356352 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚217501574371840 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10388278404096 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143005928 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10208916515328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚243785848246784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244279870 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 % _timestamp‚1.7884218175674899e+09 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101248246 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes‘áÚ.all.peak‚ 6522528073 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244279870 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10208688298496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36925463 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36924817 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244254172 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59233053 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10179288663444 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10208688298496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚217501556368896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96158518 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚216130073019796 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0K"õ÷ú&ýôM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244254172 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59233053 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10179288663444 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10208688298496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚217501556368896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96158518 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚216130073019796 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚205950767316992 trainer/global_step‚5499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244254172 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227710490887168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚243786145189376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚646 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216129832331080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚227710490887168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚5288.318205131 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101248244 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96157870 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚233397593782784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚233397569842688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227710244667392 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101248244 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚450 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚205950784356352 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚217501574371840 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10388278404096 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143005928 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10208916515328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚243785848246784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244279870 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 % _timestamp‚1.7884218175674899e+09 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101248246 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244279870 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10208688298496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36925463 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36924817 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800ÂÉÏT’2þ- _wandbruntime‚5288  _runtime‚5288±?èZÿÔ 2loss/SRqqt3/assignment_loss‚1.5493727013164638 1loss/SRqqt1/detection_loss‚2.4732350860872576 .loss/FBt1/detection_loss‚2.244013786315918 ,loss/FBt4/detection_loss‚3.0130675137043 epoch‚4 0loss/FRt2/assignment_loss‚1.5676839518547059 /loss/FRt4/assignment_loss‚3.880838378479606 0loss/FBt1/assignment_loss‚0.5174501943749116 1loss/FBt3/assignment_loss‚0.14863823226187378 /loss/FRt1/detection_loss‚2.2455317302017797 /loss/FRt3/detection_loss‚2.3877991126643288 &loss/total_loss‚43.880714843975205 /loss/FRt1/assignment_loss‚4.125116971074318 2loss/SRqqt4/assignment_loss‚1.8243557756597346 1loss/SRqqt3/detection_loss‚1.8847384763007262 $ _timestamp‚1.788421817569558e+09 _runtime‚5288.319391807 _step‚451 2loss/SRqqt2/assignment_loss‚0.8902501235501125 /loss/FRt4/detection_loss‚2.4988328587067756 .loss/FBt3/detection_loss‚2.075497397354671 /loss/FRt3/assignment_loss‚2.460222089290619 1loss/FBt2/assignment_loss‚0.17624798699317906 /loss/FRt2/detection_loss‚1.8660665113478898 1loss/SRqqt4/detection_loss‚2.0523171160708773 /loss/FBt2/detection_loss‚0.7511408018242371 2loss/SRqqt1/assignment_loss‚2.0846674423063956 1loss/FBt4/assignment_loss‚0.14948535527219065 1loss/SRqqt2/detection_loss‚1.0141452509623308 trainer/global_step‚5499Õ%%‘_€Ù trainer/global_step‚5499 2loss/SRqqt3/assignment_loss‚1.5493727013164638 1loss/SRqqt1/detection_loss‚2.4732350860872576 .loss/FBt1/detection_loss‚2.244013786315918 ,loss/FBt4/detection_loss‚3.0130675137043 epoch‚4 0loss/FRt2/assignment_loss‚1.5676839518547059 /loss/FRt4/assignment_loss‚3.880838378479606 0loss/FBt1/assignment_loss‚0.5174501943749116 1loss/FBt3/assignment_loss‚0.14863823226187378 /loss/FRt1/detection_loss‚2.2455317302017797 /loss/FRt3/detection_loss‚2.3877991126643288 &loss/total_loss‚43.880714843975205 /loss/FRt1/assignment_loss‚4.125116971074318 2loss/SRqqt4/assignment_loss‚1.8243557756597346 1loss/SRqqt3/detection_loss‚1.8847384763007262 $ _timestamp‚1.788421817569558e+09 _runtime‚5288.319391807 _step‚451 2loss/SRqqt2/assignment_loss‚0.8902501235501125 /loss/FRt4/detection_loss‚2.4988328587067756 .loss/FBt3/detection_loss‚2.075497397354671 /loss/FRt3/assignment_loss‚2.460222089290619 1loss/FBt2/assignment_loss‚0.17624798699317906 /loss/FRt2/detection_loss‚1.8660665113478898 1loss/SRqqt4/detection_loss‚2.0523171160708773 /loss/FBt2/detection_loss‚0.7511408018242371 2loss/SRqqt1/assignment_loss‚2.0846674423063956 1loss/FBt4/assignment_loss‚0.14948535527219065 1loss/SRqqt2/detection_loss‚1.0141452509623308Ë~&:‚(:„ ÀÍäÔйx gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 °ì{q‚‚(:æ ÀÍäÔ‘“°{ disk.drbd1777.out‚0 disk.dm-1.out‚ 288.703125 disk.rbd2.in‚0) proc.memory.percent‚17.47288703918457 network.recv‚ 1.871504e+06 disk.rbd4.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9765625 proc.cpu.threads‚17 network.sent‚ 1.2890438e+07% disk./.usageGB‚1098.4359092712402 disk.sda1.out‚0 disk.sda.out‚ 288.7421875 disk.dm-1.in‚7.3125 disk.rbd0.out‚0# memory_percent‚97.9511022567749% disk.nvme0c0n1.out‚23677.16015625 disk.rbd3.out‚0 disk.sda3.out‚ 288.703125# disk.nvme0n1.out‚23677.16015625 disk.dm-0.out‚23677.16015625 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125 disk.rbd1.out‚ 2.51611328125 disk.sda2.in‚0( proc.memory.availableMB‚ 335.69140625) disk./.usagePercent‚73.71695552030889 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.rbd2.out‚176.0849609375 disk.sda.in‚ 7.36328125 disk.dm-0.in‚ 4.9765625 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0" proc.memory.rssMB‚ 2862.7578125 cpu‚25.274601686568154 disk.rbd3.in‚0Ü{𠃂(:… ÏÍäÔ§’ëx gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{RØÃr„‚(:ç ÏÍäÔå¨ôz disk.rbd4.out‚0 disk.nvme0n1.in‚ 4.9765625# memory_percent‚97.9503870010376 cpu‚25.27397483755957 network.sent‚ 1.2999041e+07$ disk.nvme0c0n1.out‚ 23776.2265625 disk.sda3.out‚ 289.07421875 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125( proc.memory.availableMB‚ 335.80859375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.in‚ 0.05078125 disk.dm-0.out‚ 23776.2265625 disk.dm-1.out‚ 289.07421875 network.recv‚ 1.877715e+06% disk./.usageGB‚1098.4358520507812 disk.rbd2.in‚0" disk.nvme0n1.out‚ 23776.2265625 disk.sda2.in‚0 disk.rbd2.out‚176.0849609375 disk.drbd1777.in‚ 1.03515625 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.rbd1.out‚ 2.51611328125 disk.rbd3.out‚0) proc.memory.percent‚17.47288703918457 proc.cpu.threads‚17 disk.sda.out‚ 289.11328125 disk.dm-0.in‚ 4.9765625) disk./.usagePercent‚73.71695168019588 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0" proc.memory.rssMB‚ 2862.7578125 disk.rbd4.in‚0Cí)…‚(:… ÞÍäԃسx gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚110.784' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1568 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯€†‚(:õ ÞÍäÔ“Ðò{ network.recv‚ 1.881708e+06 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 289.63671875 disk.rbd2.in‚0 disk.rbd4.in‚0% disk./.usageGB‚1098.4362182617188 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0# disk.dm-0.out‚23875.47412109375 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9765625 network.sent‚ 1.3004968e+07 disk.sda.out‚ 289.67578125 disk.dm-1.out‚ 289.63671875! disk.rbd2.out‚176.18310546875( disk.nvme0c0n1.out‚23875.47412109375$ memory_percent‚97.95050621032715" proc.memory.rssMB‚ 2862.7578125 proc.cpu.threads‚17 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚ 4.9765625 disk.dm-0.in‚ 4.9765625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.51611328125& disk.nvme0n1.out‚23875.47412109375 disk.drbd1777.in‚ 1.03515625 cpu‚25.273337474864796 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125' proc.memory.availableMB‚ 335.7890625) disk./.usagePercent‚73.71697625691917) proc.memory.percent‚17.47288703918457¿¶‰2‡- _wandbruntime‚5333  _runtime‚5333–ätLjÁ & lr-AdamW/pg2‚0.0005264895125173878 trainer/global_step‚5549 % _timestamp‚1.7884218630198333e+09 _runtime‚5333.769828925 _step‚452 & lr-AdamW/pg1‚0.0005264895125173878`ˆmÄÌ‰Æ & lr-AdamW/pg2‚0.0005264895125173878 trainer/global_step‚5549 % _timestamp‚1.7884218630198333e+09 _runtime‚5333.769828925 _step‚452 & lr-AdamW/pg1‚0.0005264895125173878ÄŠKú÷2Š- _wandbruntime‚5333  _runtime‚5333œ$“­ë'‹åO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10298749741056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59740450 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245895075016704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229678865369088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219380115628032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235414884959232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102116475 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144286949 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚217998453937632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 _runtime‚5333.770932365 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚229678865369088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246377754 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10268866185696 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245894780157952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235414908899328 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219380133630976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207729604791296 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102116477 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 _step‚453 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96984607 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96983959 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚229679111575552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37244157 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144261279 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144261279 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 246403426 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10298977944576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚229679111575552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59740448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884218630206685e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144286949ó 1 ŒêO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59740448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884218630206685e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144286949 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10298749741056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59740450 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245895075016704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229678865369088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219380115628032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235414884959232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102116475 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144286949 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚217998453937632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 _runtime‚5333.770932365 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚229678865369088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246377754 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10268866185696 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245894780157952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235414908899328 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219380133630976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207729604791296 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102116477 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 _step‚453 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96984607 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96983959 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚229679111575552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37244157 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144261279 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144261279 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 246403426 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10298977944576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚229679111575552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360ÅA¹°’2- _wandbruntime‚5334  _runtime‚5334:Ât õ&ŽïM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207765890209792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 % _timestamp‚1.7884218639107816e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10270698585296 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _runtime‚5334.661720174 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10300587435520 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102134194 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97000524 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚245937815658496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 trainer/global_step‚5549 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102134194 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10300815652352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246446722 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59750805 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚229719287227392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246446722 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219418471575040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219418453572096 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10481761003008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144312526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚218036588795088 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚229719041007616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37250387 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37249721 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246421024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229719287227392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144312526 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144286830 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207765907249152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚235456054655488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102134196 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144286830 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102134196 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10300815652352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚219418471575040 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245938110503936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10482031908352 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219418453572096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59750803 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397´ ™bú&ôM O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219418471575040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219418453572096 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10481761003008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144312526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚218036588795088 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚229719041007616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37250387 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37249721 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246421024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229719287227392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144312526 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144286830 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207765907249152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚235456054655488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102134196 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144286830 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102134196 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10300815652352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚219418471575040 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245938110503936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10482031908352 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219418453572096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59750803 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207765890209792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 % _timestamp‚1.7884218639107816e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10270698585296 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _runtime‚5334.661720174 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10300587435520 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102134194 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97000524 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚245937815658496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 trainer/global_step‚5549 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102134194 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10300815652352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246446722 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59750805 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚229719287227392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246446722 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ƾ‘Ct2- _wandbruntime‚5334  _runtime‚5334šÓo\‘Ö 2loss/SRqqt4/assignment_loss‚2.0109545675956686 0loss/FBt3/assignment_loss‚0.1790204683318734 /loss/FRt4/detection_loss‚2.1242168165424036 0loss/SRqqt4/detection_loss‚2.139018524600112 .loss/FRt1/detection_loss‚2.590020702328793 .loss/FBt4/detection_loss‚2.581460158030192 _step‚455 1loss/FBt4/assignment_loss‚0.43626845625597827 1loss/SRqqt2/detection_loss‚0.8198203958379917 .loss/FBt3/detection_loss‚3.197544205188751 &loss/total_loss‚44.965831966732736 trainer/global_step‚5549 % _timestamp‚1.7884218639128559e+09 0loss/FRt1/assignment_loss‚3.6160243084264354 /loss/FRt4/assignment_loss‚3.373109128740099 2loss/SRqqt3/assignment_loss‚1.9378316148239023 .loss/FRt3/detection_loss‚2.625111708790064 2loss/SRqqt2/assignment_loss‚0.7268745870066758 1loss/FBt1/assignment_loss‚0.17432587388902904 /loss/FRt2/detection_loss‚1.8691306877071443 0loss/FBt2/detection_loss‚0.49392085666928826 epoch‚4 1loss/SRqqt1/detection_loss‚2.4038227260112763 _runtime‚5334.662763991 1loss/FBt2/assignment_loss‚0.13221329047411598 1loss/SRqqt3/detection_loss‚2.0460812718260524 .loss/FBt1/detection_loss‚2.204902970790863 0loss/FRt2/assignment_loss‚1.3281199647032695 0loss/FRt3/assignment_loss‚3.7252534732222555 1loss/SRqqt1/assignment_loss‚2.230785208940506VÈOa’Û trainer/global_step‚5549 % _timestamp‚1.7884218639128559e+09 0loss/FRt1/assignment_loss‚3.6160243084264354 /loss/FRt4/assignment_loss‚3.373109128740099 2loss/SRqqt3/assignment_loss‚1.9378316148239023 .loss/FRt3/detection_loss‚2.625111708790064 2loss/SRqqt2/assignment_loss‚0.7268745870066758 1loss/FBt1/assignment_loss‚0.17432587388902904 /loss/FRt2/detection_loss‚1.8691306877071443 0loss/FBt2/detection_loss‚0.49392085666928826 epoch‚4 1loss/SRqqt1/detection_loss‚2.4038227260112763 _runtime‚5334.662763991 1loss/FBt2/assignment_loss‚0.13221329047411598 1loss/SRqqt3/detection_loss‚2.0460812718260524 .loss/FBt1/detection_loss‚2.204902970790863 0loss/FRt2/assignment_loss‚1.3281199647032695 0loss/FRt3/assignment_loss‚3.7252534732222555 1loss/SRqqt1/assignment_loss‚2.230785208940506 2loss/SRqqt4/assignment_loss‚2.0109545675956686 0loss/FBt3/assignment_loss‚0.1790204683318734 /loss/FRt4/detection_loss‚2.1242168165424036 0loss/SRqqt4/detection_loss‚2.139018524600112 .loss/FRt1/detection_loss‚2.590020702328793 .loss/FBt4/detection_loss‚2.581460158030192 _step‚455 1loss/FBt4/assignment_loss‚0.43626845625597827 1loss/SRqqt2/detection_loss‚0.8198203958379917 .loss/FBt3/detection_loss‚3.197544205188751 &loss/total_loss‚44.965831966732736ǵ¬àz“‚(:„ íÍäÔŸ†µx gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.045' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.809 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßgvŽ”‚(:ö íÍäÔ‡·Í{( proc.memory.availableMB‚ 335.74609375 cpu‚25.272716060249756 disk.dm-0.in‚ 4.9765625 disk.sda3.in‚7.3125# disk.dm-0.out‚23974.56494140625 disk.dm-1.out‚ 290.19921875 disk.sda3.out‚ 290.19921875 disk.nvme0n1.in‚ 4.9765625 disk.sda2.in‚0) proc.memory.percent‚17.47288703918457 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚23974.56494140625 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 290.23828125 disk.sda2.out‚ 0.0390625 disk.sda1.in‚ 0.05078125$ memory_percent‚97.95076847076416" proc.memory.rssMB‚ 2862.7578125) disk./.usagePercent‚73.71697625691917 disk.sda.in‚ 7.36328125& disk.nvme0n1.out‚23974.56494140625! disk.rbd2.out‚176.18310546875 disk.rbd0.out‚0 network.sent‚ 1.3011305e+07 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd3.out‚0 network.recv‚ 1.886135e+06% disk./.usageGB‚1098.4362182617188 disk.rbd1.out‚ 2.66455078125 disk.dm-1.in‚7.3125 disk.sda1.out‚0ÜÏS½š•‚(: üÍäÔêõÕx gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.08' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.215999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ´ÒT:–‚(:ê üÍäÔ¨ó©{ disk.rbd2.in‚0 disk.drbd1¤o;777.in‚ 1.03515625% disk.nvme0c0n1.out‚23975.50390625 disk.rbd3.in‚0 disk.sda.out‚ 290.66015625 disk.sda2.in‚0) proc.memory.percent‚17.47288703918457 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.9765625$ memory_percent‚97.95145988464355' proc.memory.availableMB‚ 335.6328125 disk.rbd0.out‚0 disk.rbd1.out‚ 2.66455078125 disk.nvme0n1.in‚ 4.9765625 disk.sda2.out‚ 0.0390625" proc.memory.rssMB‚ 2862.7578125% disk./.usageGB‚1098.4361305236816 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 290.62109375# disk.nvme0n1.out‚23975.50390625 proc.cpu.threads‚17 network.sent‚ 1.311944e+07) disk./.usagePercent‚73.71697036874588 disk.dm-1.out‚ 290.62109375! disk.rbd2.out‚177.89794921875 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125 disk.dm-0.out‚23975.50390625 cpu‚25.27209791161342 disk.sda.in‚ 7.36328125 network.recv‚ 1.891358e+06 disk.dm-1.in‚7.3125 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 4.9765625ç ]×—‚(:… ‹ÎäÔÖÔšx gpu.0.gpu‚21.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÁ¼úc˜‚(:Ø ‹ÎäÔ–Ó±{ disk.sda1.out‚0" disk.nvme0c0n1.out‚ 24074.65625( proc.memory.availableMB‚ 335.62890625) proc.memory.percent‚17.47288703918457 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 network.recv‚ 1.896025e+06 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.dm-1.out‚291.375 disk.rbd2.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚73.71697728094931 disk.sda1.in‚ 0.05078125 disk.nvme0n1.out‚ 24074.65625 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.sda.out‚ 291.4140625 disk.sda3.out‚291.375 disk.sda2.in‚0 cpu‚25.271480041142045% disk./.usageGB‚1098.4362335205078 disk.rbd3.in‚0 disk.nvme0n1.in‚ 4.9765625 disk.dm-1.in‚7.3125" proc.memory.rssMB‚ 2862.7578125 disk.sda3.in‚7.3125 disk.dm-0.out‚ 24074.65625$ memory_percent‚97.95148372650146 disk.dm-0.in‚ 4.9765625 disk.sda2.out‚ 0.0390625! disk.rbd2.out‚177.89794921875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 2.69677734375 network.sent‚ 1.312545e+07­®d®2™- _wandbruntime‚5380  _runtime‚5380„Aò!ÆšÀ & lr-AdamW/pg1‚0.0005209753355766109 & lr-AdamW/pg2‚0.0005209753355766109 trainer/global_step‚5599 $ _timestamp‚1.788421909301319e+09 _runtime‚5380.051247546 _step‚456imȷ˛Š& lr-AdamW/pg1‚0.0005209753355766109 & lr-AdamW/pg2‚0.0005209753355766109 trainer/global_step‚5599 $ _timestamp‚1.788421909301319e+09 _runtime‚5380.051247546 _step‚456È17Ïq2œ- _wandbruntime‚5380  _runtime‚5380¢ó°ë'åO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10360499756904 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248047052813824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248046760052224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚231687661709312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103002425 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97824251 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37566683 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚231687907915776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 trainer/global_step‚5599 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103002425 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60258198 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221297030834176 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103002427 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _runtime‚5380.052181286 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚231687907915776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37566053 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10390877081600 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10390877081600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10573659101696 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10573390280192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209544727684096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209544710644736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221297030834176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60258200 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237473369772032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 % _timestamp‚1.7884219093020542e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219905451090148 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145567851 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚457 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248544606 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145542181 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221297012831232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚231687661709312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10360723406052 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219905210401640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221297012831232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145567851 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148mÆð'žêO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60258198 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221297030834176 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103002427 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _runtime‚5380.052181286 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚231687907915776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37566053 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10390877081600 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10390877081600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10573659101696 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10573390280192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209544727684096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209544710644736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221297030834176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60258200 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237473369772032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 % _timestamp‚1.7884219093020542e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219905451090148 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145567851 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚457 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248544606 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145542181 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221297012831232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚231687661709312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10360723406052 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219905210401640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221297012831232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145567851 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10360499756904 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248047052813824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248046760052224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚231687661709312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103002425 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97824251 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37566683 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚231687907915776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 trainer/global_step‚5599 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103002425É•K6)2Ÿ- _wandbruntime‚5381  _runtime‚5381±+UGƒ# ïM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 248587876 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚248090096916992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145593428 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10575533508608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚231727837347840 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145567732 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248613574 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚5381.149497332 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237514563408384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚231728083567616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10392486572544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 248587876 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10392714789376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248089793682944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97840830 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 % _timestamp‚1.7884219103988411e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103020144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103020144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237514539468288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219943345259096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 trainer/global_step‚5599 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103020146 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 248613574 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 P6D¤ÏreviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209581013102592 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221335368778240 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221335368778240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299ïDAú&¡ôM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209581013102592 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221335368778240 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221335368778240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 248587876 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚248090096916992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145593428 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10575533508608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚231727837347840 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145567732 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248613574 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚5381.149497332 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237514563408384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚231728083567616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10392486572544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 248587876 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10392714789376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248089793682944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97840830 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 % _timestamp‚1.7884219103988411e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103020144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103020144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237514539468288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219943345259096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 trainer/global_step‚5599 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103020146 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 248613574 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÊÑÝË%2¢- _wandbruntime‚5381  _runtime‚5381ЃJ|^£Ø 1loss/FBt3/assignment_loss‚0.22905775846447796 .loss/FRt4/detection_loss‚2.223920172820248 1loss/SRqqt2/detection_loss‚0.8802512575473104 .loss/FRt3/detection_loss‚2.784327413354601 /loss/FBt1/detection_loss‚2.7445550560951233 /loss/FRt1/assignment_loss‚3.821734491516562 .loss/FRt1/detection_loss‚2.709463572677444 /loss/FBt4/detection_loss‚2.1283574955804005 _step‚459 0loss/FRt3/assignment_loss‚3.2636441503252303 2loss/SRqqt3/assignment_loss‚1.6121800520409972 1loss/SRqqt4/assignment_loss‚1.813036855009775 1loss/FBt2/assignment_loss‚0.05968030791517349 /loss/FRt2/detection_loss‚1.5642521108611156 /loss/FBt2/detection_loss‚0.6731022754377735 % _timestamp‚1.7884219104007754e+09 epoch‚4 2loss/SRqqt1/assignment_loss‚1.8336747405496803 1loss/SRqqt1/detection_loss‚2.3535166592211336 1loss/SRqqt3/detection_loss‚1.7515940681352453 1loss/SRqqt4/detection_loss‚1.8604215402756967 &loss/total_loss‚43.412747493408105 0loss/FRt2/assignment_loss‚1.1893739983186884 /loss/FRt4/assignment_loss‚3.382249449193478 2loss/SRqqt2/assignment_loss‚0.7803663179234025 1loss/FBt1/assignment_loss‚0.10881222828417474 1loss/FBt4/assignment_loss‚0.20517471958185393 /loss/FBt3/detection_loss‚3.4400008022785187 trainer/global_step‚5599 _runtime‚5381.150592787¢<ïc¤Ý 1loss/FBt3/assignment_loss‚0.22905775846447796 .loss/FRt4/detection_loss‚2.223920172820248 1loss/SRqqt2/detection_loss‚0.8802512575473104 .loss/FRt3/detection_loss‚2.784327413354601 /loss/FBt1/detection_loss‚2.7445550560951233 /loss/FRt1/assignment_loss‚3.821734491516562 .loss/FRt1/detection_loss‚2.709463572677444 /loss/FBt4/detection_loss‚2.1283574955804005 _step‚459 0loss/FRt3/assignment_loss‚3.2636441503252303 2loss/SRqqt3/assignment_loss‚1.6121800520409972 1loss/SRqqt4/assignment_loss‚1.813036855009775 1loss/FBt2/assignment_loss‚0.05968030791517349 /loss/FRt2/detection_loss‚1.5642521108611156 /loss/FBt2/detection_loss‚0.6731022754377735 % _timestamp‚1.7884219104007754e+09 epoch‚4 2loss/SRqqt1/assignment_loss‚1.8336747405496803 1loss/SRqqt1/detection_loss‚2.3535166592211336 1loss/SRqqt3/detection_loss‚1.7515940681352453 1loss/SRqqt4/detection_loss‚1.8604215402756967 &loss/total_loss‚43.412747493408105 0loss/FRt2/assignment_loss‚1.1893739983186884 /loss/FRt4/assignment_loss‚3.382249449193478 2loss/SRqqt2/assignment_loss‚0.7803663179234025 1loss/FBt1/assignment_loss‚0.10881222828417474 1loss/FBt4/assignment_loss‚0.20517471958185393 /loss/FBt3/detection_loss‚3.4400008022785187 trainer/global_step‚5599 _runtime‚5381.150592787ËthŒ}¥‚(:… šÎäÔ¹®­x gpu.0.gpu‚21.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.438' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4876 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æX0tp¦‚(:å šÎäÔŽ§Öz disk.rbd2.out‚178 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚24174.24853515625# disk.dm-0.out‚24174.24853515625% disk./.usageGB‚1098.4362335205078 disk.sda2.out‚ 0.0390625 disk.rbd4.out‚0 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.dm-0.in‚ 4.9765625$ memory_percent‚97.94983863830566" proc.memory.rssMB‚ 2862.7578125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 4.9765625 cpu‚25.270873015884696 proc.cpu.threads‚17 disk.rbd1.out‚ 2.69677734375 disk.sda2.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.rbd3.in‚0' proc.memory.availableMB‚ 335.8984375) proc.memory.percent‚17.47288703918457 disk.dm-1.out‚ 292.1328125 disk.sda1.in‚ 0.05078125 network.sent‚ 1.3131348e+07 network.recv‚ 1.899998e+06 disk.rbd0.in‚0) disk./.usagePercent‚73.71697728094931 disk.sda.out‚ 292.171875 disk.sda3.in‚7.3125 disk.sda3.out‚ 292.1328125& disk.nvme0n1.out‚24174.24853515625 disk.dm-1.in‚7.3125 disk.rbd2.in‚05@{§‚(:‚ ©ÎäÔ´‰ÿw gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.03 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•;5ðp¨‚(:å ©ÎäÔ¸ÿ©{ disk.nvme0n1.in‚ 4.9765625& disk.nvme0n1.out‚24273.32666015625 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.sda2.in‚0 disk.dm-1.out‚ 293.07421875 network.sent‚ 1.3240251e+07 disk.dm-0.in‚ 4.9765625 disk.rbd0.in‚0" proc.memory.rssMB‚ 2862.7578125% disk./.usageGB‚1098.4362335205078 disk.rbd1.in‚0( disk.nvme0c0n1.out‚24273.32666015625 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.sda.out‚ 293.11328125 disk.sda3.in‚7.3125& proc.memory.availableMB‚ 336.015625 disk.sda1.out‚0 disk.rbd3.out‚0 network.recv‚ 1.906318e+06 disk.rbd1.out‚ 2.759765625 disk.rbd2.out‚178 disk.rbd4.in‚0 disk.sda.in‚ 7.36328125) proc.memory.percent‚17.47288703918457) disk./.usagePercent‚73.71697728094931 disk.drbd1777.out‚0 disk.rbd4.out‚0 cpu‚25.27025620711398# disk.dm-0.out‚24273.32666015625 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 4.9765625 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0 disk.sda3.out‚ 293.07421875$ memory_percent‚97.94912338256836ÎÆœ“©‚(:… ¸ÎäÔÀÇÈx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.867' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9734 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09%˜‡tª‚(:é ¸ÎäÔ‚ÖÊ{) disk./.usagePercent‚73.71697292882122% disk./.usageGB‚1098.4361686706543 disk.sda2.in‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.sda.out‚ 293.78125 disk.dm-1.out‚ 293.7421875 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.9765625) proc.memory.percent‚17.47288703918457 disk.rbd2.out‚ 178.140625 disk.dm-0.in‚ 4.9765625 disk.dm-1.in‚7.3125 disk.drbd1777.out‚0 disk.sda3.in‚7.3125 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 293.7421875 disk.rbd1.out‚ 2.759765625 disk.rbd2.in‚0 network.recv‚ 1.909721e+06& disk.nvme0n1.out‚24372.50634765625 disk.sda2.out‚ 0.0390625( proc.memory.availableMB‚ 335.80859375 cpu‚25.269646423176766 disk.nvme0n1.in‚ 4.9765625 disk.drbd1777.in‚ 1.03515625 disk.sda.in‚ 7.36328125# memory_percent‚97.9503870010376 network.sent‚ 1.3245567e+07# disk.dm-0.out‚24372.50634765625( disk.nvme0c0n1.out‚24372.50634765625 disk.rbd3.in‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2862.7578125¨d/2«- _wandbruntime‚5426  _runtime‚5426ǬÁ % _timestamp‚1.7884219555236714e+09 _runtime‚5426.273599168 _step‚460 & lr-AdamW/pg1‚0.0005154361891610086 & lr-AdamW/pg2‚0.0005154361891610086 trainer/global_step‚5649²£0Ì­Æ & lr-AdamW/pg1‚0.0005154361891610086 & lr-AdamW/pg2‚0.0005154361891610086 trainer/global_step‚5649 % _timestamp‚1.7884219555236714e+09 _runtime‚5426.273599168 _step‚460Ì4ý°ð2®- _wandbruntime‚5426  _runtime‚5426íguýé'¯ãO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60775950 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10452356977260 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103888377 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10482776218624 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10482776218624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10667069091328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223213928037376 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚211359850576896 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146848753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37888668 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10482548015104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221812207554156 trainer/global_step‚5649 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚239531854584832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚211359833537536 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146848753 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250711458 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚233696458049536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60775948 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103888375 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250737130 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250198947616256 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146823083 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223213928037376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250198650660352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10666796075520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 _runtime‚ 5426.2748208 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚233696704256000 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103888375 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223213910034432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚233696704256000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239531878524928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103888377 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221811966865648 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103888377 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221811966865648 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10452356977260 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103888377 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10482776218624 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10482776218624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10667069091328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223213928037376 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚211359850576896 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146848753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37888668 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10482548015104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221812207554156 trainer/global_step‚5649 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚239531854584832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚211359833537536 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146848753 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250711458 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚233696458049536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60775948 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103888375 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250737130 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250198947616256 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146823083 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0Í|†âÁ2±- _wandbruntime‚5427  _runtime‚5427–põ&²ïM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 % _timestamp‚1.7884219563992054e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 trainer/global_step‚5649 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146874330 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10484613926400 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10484613926400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98680735 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10668658308096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250754728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146848634 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103906094 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146874330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60786305 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37894432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250241682589184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221850342411820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250780426 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚233736879907840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10484385709568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚239573024281088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚5427.150062675 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚223252247978496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10484385709568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239573048221184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250780426 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223252265981440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10453965727712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103906096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60786303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚233736879907840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250754728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223252247978496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚233736633688064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221850101723104 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103906094 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10484385709568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239573048221184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250780426 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223252265981440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10453965727712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103906096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60786303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚233736879907840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250754728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223252247978496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚233736633688064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221850101723104 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103906094 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 trainer/global_step‚5649 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146874330 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10484613926400 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10484613926400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98680735 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10668658308096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250754728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146848634 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103906094 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146874330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60786305 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37894432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250241682589184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221850342411820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250780426 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚233736879907840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10484385709568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚239573024281088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚5427.150062675 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168ÎàI2´- _wandbruntime‚5427  _runtime‚5427 7‘WµÑ 2loss/SRqqt4/assignment_loss‚1.9187482771887021 .loss/FRt1/detection_loss‚2.366949533753925 /loss/FBt1/detection_loss‚2.4740286882107076 ,loss/FBt3/detection_loss‚2.5064412014825 %loss/total_loss‚44.80509121579571 -loss/SRqqt1/detection_loss‚2.345719289212 _step‚463 0loss/FRt2/assignment_loss‚1.2169563048764278 1loss/FBt1/assignment_loss‚0.08291362603911413 /loss/FRt3/detection_loss‚2.3885575253912745 1loss/SRqqt2/detection_loss‚0.9701008022721134 1loss/SRqqt4/detection_loss‚1.9992058266292918 2loss/SRqqt3/assignment_loss‚1.9705561109144112 1loss/FBt2/assignment_loss‚0.12566281242657756 .loss/FRt4/detection_loss‚2.740886544989001 epoch‚4 _runtime‚5427.151186744 /loss/FRt3/assignment_loss‚2.998462674465585 /loss/FRt2/detection_loss‚1.6867601016634388 .loss/FBt4/detection_loss‚4.021270235379537 2loss/SRqqt2/assignment_loss‚0.9018945442577789 0loss/FBt3/assignment_loss‚0.1360839811500877 /loss/FBt2/detection_loss‚0.6536437518003843 trainer/global_step‚5649 0loss/FRt1/assignment_loss‚3.2927524381213717 /loss/FRt4/assignment_loss‚4.176973665914228 2loss/SRqqt1/assignment_loss‚1.9821986030964625 2loss/FBt4/assignment_loss‚0.003506043382609884 1loss/SRqqt3/detection_loss‚1.8448186331781848 % _timestamp‚1.7884219564013062e+09Œ°;\¶Ö 0loss/FRt2/assignment_loss‚1.2169563048764278 1loss/FBt1/assignment_loss‚0.08291362603911413 /loss/FRt3/detection_loss‚2.3885575253912745 1loss/SRqqt2/detection_loss‚0.9701008022721134 1loss/SRqqt4/detection_loss‚1.9992058266292918 2loss/SRqqt3/assignment_loss‚1.9705561109144112 1loss/FBt2/assignment_loss‚0.12566281242657756 .loss/FRt4/detection_loss‚2.740886544989001 epoch‚4 _runtime‚5427.151186744 /loss/FRt3/assignment_loss‚2.998462674465585 /loss/FRt2/detection_loss‚1.6867601016634388 .loss/FBt4/detection_loss‚4.021270235379537 2loss/SRqqt2/assignment_loss‚0.9018945442577789 0loss/FBt3/assignment_loss‚0.1360839811500877 /loss/FBt2/detection_loss‚0.6536437518003843 trainer/global_step‚5649 0loss/FRt1/assignment_loss‚3.2927524381213717 /loss/FRt4/assignment_loss‚4.176973665914228 2loss/SRqqt1/assignment_loss‚1.9821986030964625 2loss/FBt4/assignment_loss‚0.003506043382609884 1loss/SRqqt3/detection_loss‚1.8448186331781848 % _timestamp‚1.7884219564013062e+09 2loss/SRqqt4/assignment_loss‚1.9187482771887021 .loss/FRt1/detection_loss‚2.366949533753925 /loss/FBt1/detection_loss‚2.4740286882107076 ,loss/FBt3/detection_loss‚2.5064412014825 %loss/total_loss‚44.80509121579571 -loss/SRqqt1/detection_loss‚2.345719289212 _step‚463Ï—GãÄ›·‚(: ÇÎäÔåÞ³x gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.956' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.191200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿û|¸‚(:ñ ÇÎäÔŒÑó{ disk.sda3.out‚ 294.19140625 disk.dm-1.in‚7.3125 disk.nvme0c0n1.in‚ 4.9765625$ memory_percent‚97.94895648956299 disk.nvme0n1.in‚ 4.9765625& disk.nvme0n1.out‚24372.80322265625 disk.rbd4.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.71697292882122% disk./.usageGB‚1098.4361686706543 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2862.7578125 network.recv‚ 1.914114e+06 disk.rbd2.out‚ 179.65234375 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 294.23046875 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd1.out‚ 2.759765625# disk.dm-0.out‚24372.80322265625 disk.dm-1.out‚ 294.19140625 disk.rbd2.in‚0( disk.nvme0c0n1.out‚24372.80322265625 network.sent‚ 1.3251581e+07 disk.rbd3.out‚0 disk.dm-0.in‚ 4.9765625( proc.memory.availableMB‚ 336.04296875) proc.memory.percent‚17.47288703918457 cpu‚25.269085451769904 disk.sda3.in‚7.3125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0/)0É›¹‚(: ÖÎäÔ¡¬¯x gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.389' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.677799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðÁѲ{º‚(:ð ÖÎäÔã±±{ disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0" proc.memory.rssMB‚ 2862.7578125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.dm-1.out‚ 294.70703125 disk.drbd1777.in‚ 1.03515625 disk.sda3.out‚ 294.70703125) disk./.usagePercent‚73.71696089646711" disk.dm-0.out‚24472.5986328125 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.sda3.in‚7.3125 disk.rbd1.out‚ 2.82275390625 disk.rbd0.out‚0 disk.rbd4.in‚0$ memory_percent‚97.94924259185791 cpu‚25.268489733266428 network.sent‚ 1.3360013e+07 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 294.74609375( proc.memory.availableMB‚ 335.99609375 network.recv‚ 1.920092e+06 disk.dm-0.in‚ 4.9765625 disk.rbd2.in‚0 disk.drbd1777.out‚0% disk./.usageGB‚1098.4359893798828 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 179.79296875 disk.sda.in‚ 7.36328125) proc.memory.percent‚17.47288703918457% disk.nvme0n1.out‚24472.5986328125 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.nvme0n1.in‚ 4.9765625' disk.nvme0c0n1.out‚24472.5986328125ÚNÕÔ»‚(:„ åÎäÔ³›¶x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.731' gpu.0.à…=‚»enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.1462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01&>8u¼‚(:ê åÎäÔìÿÿ{ disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.9765625 disk.sda1.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd4.in‚0$ memory_percent‚97.95429706573486 disk.dm-1.in‚7.3125' disk.nvme0c0n1.out‚24571.5751953125 network.recv‚ 1.924997e+06 disk.nvme0n1.in‚ 4.9765625% disk.nvme0n1.out‚24571.5751953125 cpu‚25.26788655493452 disk.dm-1.out‚ 295.109375 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125) proc.memory.percent‚17.47288703918457% disk./.usageGB‚1098.4358940124512 disk.sda3.in‚7.3125 disk.rbd1.in‚0 disk.rbd1.out‚ 2.82275390625( proc.memory.availableMB‚ 335.16796875" proc.memory.rssMB‚ 2862.7578125" disk.dm-0.out‚24571.5751953125 disk.rbd2.in‚0 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 295.109375 network.sent‚ 1.3366432e+07) disk./.usagePercent‚73.71695449627876 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 179.79296875 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚ 295.1484375 disk.dm-0.in‚ 4.9765625 B›2½- _wandbruntime‚5472  _runtime‚547297‚ƾÀ $ _timestamp‚1.788422001720948e+09 _runtime‚5472.470997161 _step‚464 & lr-AdamW/pg1‚0.0005098732276203914 & lr-AdamW/pg2‚0.0005098732276203914 trainer/global_step‚5699«5èôË¿Å & lr-AdamW/pg2‚0.0005098732276203914 trainer/global_step‚5699 $ _timestamp‚1.788422001720948e+09 _runtime‚5472.470997161 _step‚464 & lr-AdamW/pg1‚0.0005098732276203914ÐnçÓ˜2À- _wandbruntime‚5472  _runtime‚5472Püêë'ÁåO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148103985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚223718964018164 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚225130807237632 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148103985 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148129655 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235705254389760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104774325 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚5472.471915912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225130825240576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10760528051712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10574675355648 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148129655 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99506670 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚223718723329656 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235705254389760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10543990548468 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104774327 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 252878310 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10574675355648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241590339397632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38212970 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235705254389760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104774325 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚5472.471915912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225130825240576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10760528051712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10574675355648 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148129655 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99506670 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚223718723329656 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235705254389760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10543990548468 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104774327 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 252878310 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10574675355648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241590339397632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38212970 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646ÑÊ›*À2Ã- _wandbruntime‚5473  _runtime‚5473® †Ùô&ÄîM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252393923270144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚252393615841792 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10545599298920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148129536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104792046 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38219085 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788422002603537e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5699 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61304055 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚223757098875828 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10576284846592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10762106747904 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225169163184640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225169145181696 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104792046 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚466 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235745676248064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148155232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚213211275927552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99523786 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38219731 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10762390236160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104792044 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 307428352 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚213211258888192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 283488256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10545822948276 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252947278 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241631533033984 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99523138 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10576513063424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 _runtime‚5473.354047551 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648[Sð¨ÅóM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 283488256 ]JDeviceP;š›l%StatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10545822948276 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252947278 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241631533033984 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99523138 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10576513063424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 _runtime‚5473.354047551 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252393923270144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚252393615841792 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10545599298920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148129536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104792046 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38219085 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788422002603537e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5699 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61304055 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚223757098875828 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10576284846592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10762106747904 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225169163184640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225169145181696 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104792046 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚466 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235745676248064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148155232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚213211275927552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99523786 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38219731 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10762390236160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104792044 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 307428352 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚213211258888192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716ÒV2Æ- _wandbruntime‚5473  _runtime‚5473S‰{7^ÇØ epoch‚4 0loss/FRt3/assignment_loss‚3.1212442982487563 1loss/FBt1/assignment_loss‚0.11323027338784998 0loss/FBt3/assignment_loss‚0.1908598368987441 /loss/FRt1/detection_loss‚2.7116102129220963 1loss/SRqqt3/detection_loss‚1.8768216800689697 /loss/FRt4/detection_loss‚2.3247915533455963 0loss/SRqqt1/detection_loss‚2.647133391102155 /loss/FRt2/detection_loss‚1.9053764652150373 2loss/SRqqt1/assignment_loss‚2.0958373924096425 1loss/SRqqt2/detection_loss‚0.8258995585557487 1loss/SRqqt4/detection_loss‚1.8723921719587073 2loss/SRqqt4/assignment_loss‚1.5736083133181311 % _timestamp‚1.7884220026052482e+09 /loss/FBt2/detection_loss‚0.5118875741084785 /loss/FBt4/detection_loss‚2.7451680541038512 trainer/global_step‚5699 /loss/FRt1/assignment_loss‚3.946244355794546 2loss/SRqqt2/assignment_loss‚0.9620557261967204 /loss/FBt1/detection_loss‚3.8887337190764293 /loss/FRt4/assignment_loss‚3.800029369917783 1loss/FBt4/assignment_loss‚0.28149113412946464 0loss/FBt2/assignment_loss‚0.1578345782033469 .loss/FRt3/detection_loss‚2.546097152843708 /loss/FBt3/detection_loss‚2.8018442034721374 &loss/total_loss‚46.781411616578296 _runtime‚5473.355073405 _step‚467 0loss/FRt2/assignment_loss‚1.9578828538457553 2loss/SRqqt3/assignment_loss‚1.9233377474546431Å1ÈDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚469 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243648848150528 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105660275 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10666574492672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 % _timestamp‚1.7884220481938837e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚237714050729984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 trainer/global_step‚5749 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 255045162 _runtime‚5518.944008897 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10666574492672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38538618 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105660277 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149384887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61811450 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10666346289152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38537948 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10635624119676 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚237714050729984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚214990096362496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149410557 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚237714296936448 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚227047722443776 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61811448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100349396 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254502825862656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚254502528906752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149384887 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚237714296936448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262ÂÔr,ÔìO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚237714050729984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 trainer/global_step‚5749 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 255045162 _runtime‚5518.944008897 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10666574492672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38538618 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105660277 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149384887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61811450 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10666346289152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38537948 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10635624119676 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚237714050729984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚214990096362496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149410557 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚237714296936448 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚227047722443776 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61811448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100349396 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254502825862656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚254502528906752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149384887 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚237714296936448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255070834 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100350068 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_÷°‘Æbytes.small_pool.allocated‚10853977712128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚469 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243648848150528 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105660275 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10666574492672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 % _timestamp‚1.7884220481938837e+09ÕÈLêÌ2Õ- _wandbruntime‚5519  _runtime‚5519ÙOã÷&ÖñM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 _step‚470 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10637232870128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105677994 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100366509 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10855572772352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105677996 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105677994 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149436134 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149410438 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100367157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚243690017846784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚225663614651120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255088432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38545352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254545566679040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 % _timestamp‚1.7884220490756168e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255114130 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61821805 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10668412200448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105677996 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚243689993906688 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10668183983616 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚227086060387840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10855843677696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255114130 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215026398820352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚237754226368512 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255088432 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149436134 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149410438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚646 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38544706 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10637456519484 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚237754472588288 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚227086042384896 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 _runtime‚5519.826584184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215026381780992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚5749 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105677996 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚243689993906688 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10668183983616 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚227086060387840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10855843677696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255114130 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215026398820352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚237754226368512 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255088432 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149436134 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149410438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚646 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38544706 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10637456519484 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚237754472588288 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚227086042384896 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 _runtime‚5519.826584184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215026381780992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚5749 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10637232870128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105677994 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100366509 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10855572772352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105677996 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105677994 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149436134 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149410438 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100367157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚243690017846784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚225663614651120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255088432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38545352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254545566679040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 % _timestamp‚1.7884220490756168e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255114130 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61821805 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10668412200448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262ÖJ¸¦²2Ø- _wandbruntime‚5519  _runtime‚5519Ħ‡YÙÓ 1loss/SRqqt3/detection_loss‚1.8846816081626743 /loss/FBt1/detection_loss‚2.1868240892887116 0loss/FRt3/assignment_loss‚3.8009572715470283 2loss/SRqqt4/assignment_loss‚1.6468403280033903 0loss/FBt3/assignment_loss‚0.4294466064311564 0loss/FBt4/assignment_loss‚0.2950960189548575 .loss/FRt2/detection_loss‚1.659654261604432 $ _timestamp‚1.788422049077963e+09 1loss/SRqqt1/detection_loss‚2.3454518037182943 /loss/FBt2/detection_loss‚0.7384411342396044 trainer/global_step‚5749 _runtime‚5519.827847384 _step‚471 2loss/SRqqt1/assignment_loss‚1.7637542294604438 2loss/SRqqt3/assignment_loss‚1.4420677405946396 0loss/FBt1/assignment_loss‚0.4261648309417069 /loss/FRt3/detection_loss‚2.9988684428460672 /loss/FBt3/detection_loss‚3.4481422901153564 epoch‚4 .loss/FRt2/assignment_loss‚1.14365313802996 0loss/FRt4/assignment_loss‚3.1089648008346558 2loss/SRqqt2/assignment_loss‚1.0667966602902328 /loss/FRt1/detection_loss‚2.6198194718195333 /loss/FRt4/detection_loss‚2.5195732131236936 /loss/FRt1/assignment_loss‚4.078876713911693 0loss/FBt2/assignment_loss‚0.1721933408839971 0loss/SRqqt4/detection_loss‚1.774524468486592 %loss/total_loss‚44.62649327291146 1loss/SRqqt2/detection_loss‚0.8922900540101064 /loss/FBt4/detection_loss‚2.1834107556126336ÃÕ9^ÚØ 2loss/SRqqt4/assignment_loss‚1.6468403280033903 0loss/FBt3/assignment_loss‚0.4294466064311564 0loss/FBt4/assignment_loss‚0.2950960189548575 .loss/FRt2/detection_loss‚1.659654261604432 $ _timestamp‚1.788422049077963e+09 1loss/SRqqt1/detection_loss‚2.3454518037182943 /loss/FBt2/detection_loss‚0.7384411342396044 trainer/global_step‚5749 _runtime‚5519.827847384 _step‚471 2loss/SRqqt1/assignment_loss‚1.7637542294604438 2loss/SRqqt3/assignment_loss‚1.4420677405946396 0loss/FBt1/assignment_loss‚0.4261648309417069 /loss/FRt3/detection_loss‚2.9988684428460672 /loss/FBt3/detection_loss‚3.4481422901153564 epoch‚4 .loss/FRt2/assignment_loss‚1.14365313802996 0loss/FRt4/assignment_loss‚3.1089648008346558 2loss/SRqqt2/assignment_loss‚1.0667966602902328 /loss/FRt1/detection_loss‚2.6198194718195333 /loss/FRt4/detection_loss‚2.5195732131236936 /loss/FRt1/assignment_loss‚4.078876713911693 0loss/FBt2/assignment_loss‚0.1721933408839971 0loss/SRqqt4/detection_loss‚1.774524468486592 %loss/total_loss‚44.62649327291146 1loss/SRqqt2/detection_loss‚0.8922900540101064 /loss/FBt4/detection_loss‚2.1834107556126336 1loss/SRqqt3/detection_loss‚1.8846816081626743 /loss/FBt1/detection_loss‚2.1868240892887116 0loss/FRt3/assignment_loss‚3.8009572715470283×m'éÿÛ‚(:… ¡ÏäÔß…¬x gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.903' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7806 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îw î‚Ü‚(:÷ ¡ÏäÔƹ³{ disk.sda2.in‚0 disk.dm-1.in‚7.3125! disk.rbd2.out‚181.54736328125 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.dm-0.in‚ 4.98046875( disk.nvme0c0n1.out‚24869.78564453125 disk.rbd3.in‚0 disk.sda.out‚ 297.80859375$ memory_percent‚97.95024394989014( proc.memory.availableMB‚ 335.83203125 disk.rbd2.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2862.7578125 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚ 4.98046875 disk.sda3.in‚7.3125 network.sent‚ 1.349192e+07$ disk./.usageGB‚1098.436393737793 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625# disk.dm-0.out‚24869.78564453125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.98046875 cpu‚25.265535151909663 network.recv‚ 1.943216e+06 disk.dm-1.out‚ 297.76953125 disk.drbd1777.out‚0 disk.sda3.out‚ 297.76953125 disk.rbd1.in‚0 disk.rbd1.out‚ 2.82275390625) proc.memory.percent‚17.47288703918457 proc.cpu.threads‚17) disk./.usagePercent‚73.71698803326574 disk.rbd4.out‚0& disk.nvme0n1.out‚24869.78564453125GkÉÝ‚(:… °ÏäÔʳx gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.073' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.6146 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø‚Þ‚(:÷ °ÏäÔ··êz disk.dm-1.out‚ 298.30078125 cpu‚25.264959167989517 disk.sda.out‚ 298.33984375 disk.nvme0n1.in‚ 4.98046875 disk.rbd4.in‚0$ disk./.usageGB‚1098.436393737793( disk.nvme0c0n1.out‚24969.42822265625 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0" proc.memory.rssMB‚ 2862.7578125 disk.nvme0c0n1.in‚ 4.98046875 disk.sda3.in‚7.3125 disk.sda3.out‚ 298.30078125 disk.drbd1777.out‚0' proc.memory.availableMB‚ 335.5859375 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0& disk.nvme0n1.out‚24969.42822265625 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.71698803326574 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd2.in‚0) proc.memory.percent‚17.47288703918457 disk.sda.in‚ 7.36328125 disk.dm-0.in‚ 4.98046875# disk.dm-0.out‚24969.42822265625! disk.rbd2.out‚181.67236328125 network.sent‚ 1.3600049e+07 network.recv‚ 1.948948e+06 disk.rbd3.in‚0 disk.rbd1.in‚0$ memory_percent‚97.95174598693848 disk.rbd1.out‚ 2.82275390625 disk.sda2.in‚0@µ\¹ß‚(:„ ¿ÏäÔÇ«íx gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.201' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ ]à‚(:÷ ¿ÏäÔÆ”Ð{ disk.nvme0c0n1.in‚ 4.98046875 disk.rbd1.out‚ 2.82275390625& disk.nvme0n1.out‚25068.46728515625# disk.dm-0.out‚25068.46728515625( proc.memory.availableMB‚ 335.83203125" proc.memory.rssMB‚ 2862.7578125 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 network.sent‚ 1.3606229e+07! disk.rbd2.out‚181.68798828125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚25068.46728515625$ memory_percent‚97.95024394989014) proc.memory.percent‚17.47288703918457 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 298.74609375 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚ 4.98046875 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 298.74609375 disk.sda.out‚ 298.78515625 disk.dm-0.in‚ 4.98046875 network.recv‚ 1.953518e+06$ disk./.usageGB‚1098.436393737793 disk.rbd2.in‚0 disk.sda1.in‚ Ê]Ã3d0.05078125 cpu‚25.26437344583205) disk./.usagePercent‚73.71698803326574 disk.rbd4.in‚0ý~µÌá‚(:… ÎÏäÔðÏ x gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚110.637' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;Gedâ‚(:ö ÎÏäÔÇïÁz disk.sda.out‚ 299.61328125 disk.sda2.out‚ 0.0390625 disk.sda1.in‚ 0.05078125) proc.memory.percent‚17.47288703918457 disk.sda3.in‚7.3125 disk.rbd1.in‚0 disk.rbd1.out‚ 2.82275390625 disk.nvme0n1.in‚ 4.98046875 disk.rbd3.in‚0$ memory_percent‚97.95193672180176" proc.memory.rssMB‚ 2862.7578125% disk./.usageGB‚1098.1765823364258& disk.nvme0n1.out‚25069.23291015625# disk.dm-0.out‚25069.23291015625 proc.cpu.threads‚17 disk.sda.in‚ 7.36328125 disk.dm-0.in‚ 4.98046875 disk.rbd4.out‚0 disk.sda1.out‚0 cpu‚25.263808849398384( disk./.usagePercent‚73.6995518721237 disk.sda3.out‚ 299.57421875 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.98046875 disk.sda2.in‚0 disk.rbd0.in‚0 network.sent‚ 1.3612074e+07 network.recv‚ 1.957602e+06 disk.dm-1.in‚7.3125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚25069.23291015625 disk.rbd3.out‚0 disk.dm-1.out‚ 299.57421875 disk.rbd2.out‚181.8369140625 disk.rbd4.in‚0' proc.memory.availableMB‚ 335.5546875¹¡O×2ã- _wandbruntime‚5565  _runtime‚5565¡ÌkÇäÁ & lr-AdamW/pg1‚0.0004986805011369386 & lr-AdamW/pg2‚0.0004986805011369386 trainer/global_step‚5799 % _timestamp‚1.7884220947046528e+09 _runtime‚5565.454574399 _step‚472êÙLpÌåÆ & lr-AdamW/pg2‚0.0004986805011369386 trainer/global_step‚5799 % _timestamp‚1.7884220947046528e+09 _runtime‚5565.454574399 _step‚472 & lr-AdamW/pg1‚0.0004986805011369386Ø%8ä2æ- _wandbruntime‚5565  _runtime‚5565yú¸ì'çæO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10758245426176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10947541378048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5565.455453245 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62329198 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38863480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10758245426176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256654575288320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚245707309023232 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚473 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257212014 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257237686 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150665789 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5799 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚245707332963328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚227532476946180 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚239722847070208 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚216805219255296 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150665789 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106546225 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106546227 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150691459 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10947541378048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5565.455453245 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62329198 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38863480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10758245426176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256654575288320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚245707309023232 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚473 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257212014 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257237686 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150665789 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5799 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚245707332963328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚227532476946180 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚239722847070208 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚216805219255296 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150665789 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106546225 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106546227 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150691459 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106563946 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10729090090692 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150717036 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257280982 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚245748478719488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229002939588096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38870336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚245748502659584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62339553 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256697614751744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚239763022708736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256697911694336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150717036 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257280982 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚239763268928512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10728866441336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257255284 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10760083120640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106563944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚239763268928512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229002957591040 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101209233 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150691340 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 _step‚474 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229002957591040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227570371115128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101209891 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216841521713152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216841504673792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106563944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106563946 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10729090090692 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150717036 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257280982 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚245748478719488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229002939588096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38870336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚245748502659584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62339553 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256697614751744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚239763022708736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256697911694336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150717036 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257280982 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚239763268928512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10728866441336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257255284 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10760083120640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106563944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚239763268928512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229002957591040 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101209233 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150691340 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 _step‚474 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229002957591040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227570371115128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101209891 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216841521713152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216841504673792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106563944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592Ú,bZ2ì- _wandbruntime‚5566  _runtime‚5566ÍF0ƒ\íÖ % _timestamp‚1.7884220956094697e+09 0loss/FRt2/assignment_loss‚1.3701222814046419 1loss/FBt1/assignment_loss‚0.26363975687239033 epoch‚4 2loss/SRqqt1/assignment_loss‚2.1551916253182197 trainer/global_step‚5799 0loss/FRt4/assignment_loss‚3.3246710882630457 1loss/SRqqt3/detection_loss‚1.8985764011740685 %loss/total_loss‚43.79601774327482 _runtime‚5566.359304932 _step‚475 0loss/FRt3/assignment_loss‚3.1891872189482866 2loss/SRqqt3/assignment_loss‚1.8210901951847167 /loss/FRt1/detection_loss‚2.8144465701920645 /loss/FRt3/detection_loss‚2.1841943105872796 0loss/SRqqt4/detection_loss‚1.969687905576494 2loss/SRqqt2/assignment_loss‚0.9884767674332993 0loss/FBt3/assignment_loss‚0.5993328609038144 1loss/FBt4/assignment_loss‚0.07761765085160732 1loss/SRqqt2/detection_loss‚0.9979570710523562 .loss/FBt1/detection_loss‚2.159355424917661 /loss/FBt2/detection_loss‚0.5409275224151951 .loss/FBt4/detection_loss‚3.237292230129242 2loss/SRqqt4/assignment_loss‚1.5250710467497508 1loss/FBt2/assignment_loss‚0.12053119511559668 /loss/FRt4/detection_loss‚2.3302596571834466 0loss/SRqqt1/detection_loss‚2.433685087388562 /loss/FBt3/detection_loss‚2.8415397235325406 .loss/FRt1/assignment_loss‚3.35756881577628 /loss/FRt2/detection_loss‚1.59559533630426122êšqaîÛ 1loss/FBt4/assignment_loss‚0.07761765085160732 1loss/SRqqt2/detection_loss‚0.9979570710523562 .loss/FBt1/detection_loss‚2.159355424917661 /loss/FBt2/detection_loss‚0.5409275224151951 .loss/FBt4/detection_loss‚3.237292230129242 2loss/SRqqt4/assignment_loss‚1.5250710467497508 1loss/FBt2/assignment_loss‚0.12053119511559668 /loss/FRt4/detection_loss‚2.3302596571834466 0loss/SRqqt1/detection_loss‚2.433685087388562 /loss/FBt3/detection_loss‚2.8415397235325406 .loss/FRt1/assignment_loss‚3.35756881577628 /loss/FRt2/detection_loss‚1.5955953363042612 % _timestamp‚1.7884220956094697e+09 0loss/FRt2/assignment_loss‚1.3701222814046419 1loss/FBt1/assignment_loss‚0.26363975687239033 epoch‚4 2loss/SRqqt1/assignment_loss‚2.1551916253182197 trainer/global_step‚5799 0loss/FRt4/assignment_loss‚3.3246710882630457 1loss/SRqqt3/detection_loss‚1.8985764011740685 %loss/total_loss‚43.79601774327482 _runtime‚5566.359304932 _step‚475 0loss/FRt3/assignment_loss‚3.1891872189482866 2loss/SRqqt3/assignment_loss‚1.8210901951847167 /loss/FRt1/detection_loss‚2.8144465701920645 /loss/FRt3/detection_loss‚2.1841943105872796 0loss/SRqqt4/detection_loss‚1.969687905576494 2loss/SRqqt2/assignment_loss‚0.9884767674332993 0loss/FBt3/assignment_loss‚0.5993328609038144Û;˜¦›ï‚(: ÝÏäÔ²˜x gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘羃ð‚(:ø ÝÏäÔ“àÂz disk.rbd2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.64337025071207 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625( proc.memory.availableMB‚ 335.55859375) proc.memory.percent‚17.47288703918457 disk.rbd4.in‚0 disk.sda3.out‚ 300.45703125 disk.rbd4.out‚0 disk.rbd3.out‚0 cpu‚25.263236755282303 network.recv‚ 1.962334e+06 disk.rbd1.out‚ 2.82275390625 disk.dm-1.out‚ 300.45703125 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚25334.60400390625 disk.sda.in‚ 7.36328125 disk.sda.out‚ 300.49609375& disk.nvme0n1.out‚25334.60400390625 disk.sda2.out‚ 0.0390625 disk.dm-0.in‚ 4.98046875 disk.nvme0c0n1.in‚ 4.98046875 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd3.in‚0$ memory_percent‚97.95191287994385% disk./.usageGB‚1097.3394355773926 disk.nvme0n1.in‚ 4.98046875 disk.rbd2.out‚181.8486328125 disk.sda3.in‚7.3125" proc.memory.rssMB‚ 2862.7578125 network.sent‚ 1.3618429e+07 disk.rbd1.in‚0# disk.dm-0.out‚25334.60400390625 disk.sda2.in‚0„°3$ñ‚(:… ìÏäÔî”x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UhÛrò‚(:ç ìÏäÔÐçÄz disk.rbd3.in‚0 disk.dm-1.out‚ 301.23828125 disk.rbd4.in‚0% disk./.usageGB‚1097.3394050598145 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.dm-0.in‚ 4.98046875 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 301.27734375 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.47288703918457 network.recv‚ 1.968129e+06! disk.nvme0n1.out‚ 25335.421875 disk.rbd2.out‚183.4228515625 disk.sda3.in‚7.3125 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 cpu‚25.262667601568747 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.98046875' proc.memory.availableMB‚ 336.0078125 network.sent‚ 1.3726415e+07( disk./.usagePercent‚73.6433682026518 disk.sda3.out‚ 301.23828125 disk.nvme0n1.in‚ 4.98046875 disk.dm-0.out‚ 25335.421875 disk.sda2.out‚ 0.0390625 disk.sda1.out‚0$ memory_percent‚97.94917106628418 disk.rbd1.out‚ 2.82275390625# disk.nvme0c0n1.out‚ 25335.421875 disk.sda1.in‚ 0.05078125" proc.memory.rssMB‚ 2862.7578125ÝftÝŽó‚(:ƒ ûÏäÔ²æ°x gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.215' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.043 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ„‚|oô‚(:ä ûÏäÔ çÀz) disk./.usagePercent‚73.64331264901686 disk.nvme0c0n1.in‚ 4.98046875 disk.nvme0n1.in‚ 4.98046875 disk.rbd2.in‚0 disk.rbd2.out‚183.4384765625 disk.sda1.in‚ 0.05078125 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚97.95076847076416( proc.memory.availableMB‚ 335.74609375 network.sent‚ 1.3732403e+07 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.rbd1.out‚ 2.82275390625 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625! disk.nvme0c0n1.out‚ 25335.6875 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 4.98046875 proc.cpu.threads‚17 network.recv‚ 1.972645e+06 disk.dm-0.out‚ 25335.6875 disk.dm-1.in‚7.3125% disk./.usageGB‚1097.3385772705078 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.sda2.out‚ 0.0390625# proc.memory.rssMB‚ 2862.76171875 cpu‚25.262101952757376 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda.out‚ 301.61328125 disk.dm-1.out‚ 301.57421875) proc.memory.percent‚17.47291088104248 disk.drbd1777.out‚0 disk.sda3.out‚ 301.57421875 disk.nvme0n1.out‚ 25335.6875ê¿ÚŽ2õ- _wandbruntime‚5611  _runtime‚5611žû«ÐÅö¿ % lr-AdamW/pg2‚0.000493053068741526 trainer/global_step‚5849 % _timestamp‚1.7884221409363883e+09 _runtime‚5611.686276102 _step‚476 % lr-AdamW/pg1‚0.0004930530687415266+M"Ê÷Ä _runtime‚5611.686276102 _step‚476 % lr-AdamW/pg1‚0.000493053068741526 % lr-AdamW/pg2‚0.000493053068741526 trainer/global_step‚5849 % _timestamp‚1.7884221409363883e+09ÜhK–ð2ø- _wandbruntime‚5611  _runtime‚5611å]ßí'ùçO N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102032290 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218620342148096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 _runtime‚5611.687157152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39185342 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151946691 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚241731643410432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10850372766720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10850144563200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚241731643410432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62846950 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151946691 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218620325108736 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107432175_É7äúìO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚241731643410432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62846950 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151946691 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218620325108736 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107432175 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102032290 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218620342148096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 _runtime‚5611.687157152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39185342 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151946691 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚241731643410432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10850372766720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10850144563200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561Ýl ¯Œ2û- _wandbruntime‚5612  _runtime‚5612hÅò²÷&üñM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102049275 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11042805411328 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10852210474496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218656644605952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151972242 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 264613888 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚230919836791296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229477127579136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258849792883712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 259447834 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚230919854794240 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102049901 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39192596 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 288553984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218656627566592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10852210474496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11042540797440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚478 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 259422136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 trainer/global_step‚5849 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10851982257664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚230919836791296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259447834 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259422136 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107449894 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247806963532288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62857305 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10820500012544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚258849504329728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚241772065268736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚247806987472384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107449896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229477368267852 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10820723661900 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62857303 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚230919836791296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229477127579136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258849792883712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 259447834 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚230919854794240 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102049901 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39192596 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 288553984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218656627566592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10852210474496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11042540797440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚478 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 259422136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 trainer/global_step‚5849 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10851982257664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚230919836791296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259447834 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259422136 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107449894 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247806963532288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62857305 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10820500012544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚258849504329728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚241772065268736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚247806987472384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107449896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229477368267852 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10820723661900 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62857303 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102049275 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11042805411328 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10852210474496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218656644605952Þð¹S2þ- _wandbruntime‚5612  _runtime‚5612ŠOY]ÿ× 0loss/SRqqt4/detection_loss‚1.844110867724969 /loss/FRt4/assignment_loss‚3.500624187967994 1loss/SRqqt4/assignment_loss‚1.512157120491163 0loss/FBt2/assignment_loss‚0.1339720331427703 2loss/FBt3/assignment_loss‚0.006264274108356663 1loss/SRqqt3/detection_loss‚1.8590369714157922 0loss/FRt2/assignment_loss‚1.2652997993073374 2loss/SRqqt2/assignment_loss‚0.7904313654263517 2loss/SRqqt3/assignment_loss‚1.5574931235252214 /loss/FRt3/detection_loss‚2.6206805502496113 1loss/SRqqt2/detection_loss‚0.9420065252638575 /loss/FBt2/detection_loss‚0.5742576213760509 epoch‚4 /loss/FRt2/detection_loss‚1.7746638063130513 /loss/FRt4/detection_loss‚2.4237157241864637 &loss/total_loss‚43.978222769868495 trainer/global_step‚5849 1loss/FBt4/assignment_loss‚0.13876731629716232 .loss/FBt1/detection_loss‚3.065985153118769 $ _timestamp‚1.788422141806834e+09 _runtime‚5612.556637815 0loss/FRt3/assignment_loss‚3.3826705420889507 2loss/SRqqt1/assignment_loss‚1.9390554250897587 /loss/FBt3/detection_loss‚2.8558161769594466 1loss/FBt1/assignment_loss‚0.31567754798258346 0loss/SRqqt1/detection_loss‚2.250228894723428 _step‚479 /loss/FRt1/assignment_loss‚3.796844814504896 /loss/FRt1/detection_loss‚2.5191882611030625 /loss/FBt4/detection_loss‚2.9092746675014496V¡/b€Ü 1loss/FBt1/assignment_loss‚0.31567754798258346 0loss/SRqqt1/detection_loss‚2.250228894723428 _step‚479 /loss/FRt1/assignment_loss‚3.796844814504896 /loss/FRt1/detection_loss‚2.5191882611030625 /loss/FBt4/detection_loss‚2.9092746675014496 0loss/SRqqt4/detection_loss‚1.844110867724969 /loss/FRt4/assignment_loss‚3.500624187967994 1loss/SRqqt4/assignment_loss‚1.512157120491163 0loss/FBt2/assignment_loss‚0.1339720331427703 2loss/FBt3/assignment_loss‚0.006264274108356663 1loss/SRqqt3/detection_loss‚1.8590369714157922 0loss/FRt2/assignment_loss‚1.2652997993073374 2loss/SRqqt2/assignment_loss‚0.7904313654263517 2loss/SRqqt3/assignment_loss‚1.5574931235252214 /loss/FRt3/detection_loss‚2.6206805502496113 1loss/SRqqt2/detection_loss‚0.9420065252638575 /loss/FBt2/detection_loss‚0.5742576213760509 epoch‚4 /loss/FRt2/detection_loss‚1.7746638063130513 /loss/FRt4/detection_loss‚2.4237157241864637 &loss/total_loss‚43.978222769868495 trainer/global_step‚5849 1loss/FBt4/assignment_loss‚0.13876731629716232 .loss/FBt1/detection_loss‚3.065985153118769 $ _timestamp‚1.788422141806834e+09 _runtime‚5612.556637815 0loss/FRt3/assignment_loss‚3.3826705420889507 2loss/SRqqt1/assignment_loss‚1.9390554250897587 /loss/FBt3/detection_loss‚2.8558161769594466ßÍ°L‚(:… ŠÐäÔÚŠx gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®¼¥u‚‚(:ê ŠÐäÔ³áÛ{ disk.rbd2.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 4.98046875 disk.sda1.out‚0$ memory_percent‚97.95088768005371) proc.memory.percent‚17.47291088104248" disk.dm-0.out‚25336.6728515625 disk.sda2.in‚0 network.recv‚ 1.976854e+06% disk./.usageGB‚1097.3385772705078 disk.rbd4.out‚0' proc.memory.availableMB‚ 335.7265625 disk.rbd3.in‚0 disk.dm-1.out‚302.125 disk.rbd2.out‚183.5634765625# proc.memory.rssMB‚ 2862.76171875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.98046875 disk.sda3.out‚302.125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda.out‚ 302.1640625 cpu‚25.261527804786308 proc.cpu.threads‚17 disk.rbd1.out‚ 2.82275390625 network.sent‚ 1.3738329e+07) disk./.usagePercent‚73.64331264901686 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625' disk.nvme0c0n1.out‚25336.6728515625 disk.nvme0n1.in‚ 4.98046875% disk.nvme0n1.out‚25336.6728515625–÷ˆƒ‚(:… ™ÐäÔüáx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.687' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1374 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{£|„‚(:ñ ™ÐäÔ¸ÆÆ{ disk.rbd4.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚183.5791015625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 302.69140625 disk.dm-0.in‚ 4.98046875 disk.sda3.out‚ 302.65234375 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.98046875 disk.nvme0n1.in‚ 4.98046875 network.recv‚ 1.983543e+06# disk./.usageGB‚1097.33882522583& disk.nvme0c0n1.out‚25336.990234375 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0! disk.dm-0.out‚25336.990234375 disk.drbd1777.in‚ 1.03515625 disk.sda.in‚ 7.36328125$ disk.nvme0n1.out‚25336.990234375( proc.memory.availableMB‚ 335.48046875 network.sent‚ 1.3847375e+07) disk./.usagePercent‚73.64332928950658 disk.rbd1.out‚ 2.82275390625$ memory_percent‚97.95238971710205 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚7.3125# proc.memory.rssMB‚ 2862.76171875) proc.memory.percent‚17.47291088104248 cpu‚25.260969398761908 proc.cpu.threads‚17 disk.dm-1.in‚7.3125 disk.rbd1.in‚0 disk.dm-1.out‚ 302.65234375êuåA…‚(:… ¨ÐäÔàʼnx gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.158' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çž¾áà†‚(:ù ¨ÐäÔ»÷üz# proc.memory.rssMB‚ 2862.76171875 proc.cpu.threads‚17 disk.rbd1.out‚ 2.82275390625 network.recv‚ 1.987034e+06 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.98046875& disk.nvme0n1.out‚25337.43896484375 disk.sda.out‚ 303.48046875# disk.dm-0.out‚25337.43896484375) proc.memory.percent‚17.47291088104248 network.sent‚ 1.3852757e+07 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 303.44140625 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚ 4.98046875( disk.nvme0c0n1.out‚25337.43896484375' proc.memory.availableMB‚ 335.4765625 disk.sda3.in‚7.3125! disk.rbd2.out‚183.74365234375% disk./.usageGB‚1097.3383407592773 disk.rbd3.out‚Ƕ"ç¤0 disk.rbd1.in‚0$ memory_percent‚97.95241355895996 disk.rbd3.in‚0 disk.sda3.out‚ 303.44140625) disk./.usagePercent‚73.64329677654973 disk.sda1.in‚ 0.05078125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 cpu‚25.260417756812718 disk.sda.in‚ 7.36328125 disk.dm-0.in‚ 4.98046875 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.drbd1777.out‚0ÿë¢2‡- _wandbruntime‚5657  _runtime‚5657ÛSö„ƈÀ % _timestamp‚1.7884221870557191e+09 _runtime‚ 5657.80565038 _step‚480 & lr-AdamW/pg1‚0.0004874064858298454 & lr-AdamW/pg2‚0.0004874064858298454 trainer/global_step‚5899}ÔˉŠ& lr-AdamW/pg1‚0.0004874064858298454 & lr-AdamW/pg2‚0.0004874064858298454 trainer/global_step‚5899 % _timestamp‚1.7884221870557191e+09 _runtime‚ 5657.80565038 _step‚480à}în2Š- _wandbruntime‚5657  _runtime‚5657vµì'‹æO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _step‚481 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39510067 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260958342451712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 $ _timestamp‚1.788422187056459e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108318125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚243740685957120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚243740439750656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10942271903744 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108318127 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232798396050432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153253263 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307441664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚231345749185688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚249824278648832 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102874092 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63364700 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚232798396050432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 283501568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153227593 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249824302588928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220435448001536 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚243740685957120 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108318125 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10910524833300 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108318127 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232798396050432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 307470848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153253263 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307441664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚231345749185688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚249824278648832 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102874092 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63364700 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚232798396050432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 283501568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153227593 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249824302588928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220435448001536 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚243740685957120 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108318125 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10910524833300 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260958342451712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 $ _timestamp‚1.788422187056459e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108318125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚243740685957120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚243740439750656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10942271903744 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 261588988 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102891365 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63375053 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39516312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63375055 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚261001405395456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5899 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102892015 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10944109611520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 261614686 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10943881394688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153278840 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232836733994496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 261614686 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10912133583752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39516960 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220471750459392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292748288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚231384124731860 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108335844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚243780861608960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153253144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚249865472285184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220471767498752 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚231383884043144 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚243780615389184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153278840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0è/SöM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108335846 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10944109611520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚249865448345088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11136225858560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294 ”žßá!673 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 261588988 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102891365 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63375053 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39516312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63375055 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚261001405395456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5899 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102892015 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10944109611520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 261614686 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10943881394688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153278840 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232836733994496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 261614686 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10912133583752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39516960 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220471750459392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292748288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚231384124731860 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108335844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚243780861608960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153253144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚249865472285184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220471767498752 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚231383884043144 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚243780615389184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153278840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚232836751997440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 261588988âÉ›W~2- _wandbruntime‚5658  _runtime‚5658ÀÆá&[‘Õ /loss/FRt1/assignment_loss‚3.989245746816908 1loss/FBt2/assignment_loss‚0.10427682638186869 0loss/FBt4/assignment_loss‚0.2660504922802959 .loss/FRt3/detection_loss‚2.367397218233063 0loss/SRqqt1/detection_loss‚2.282079409393999 _step‚483 /loss/FRt4/detection_loss‚2.2870238399505642 trainer/global_step‚5899 1loss/SRqqt1/assignment_loss‚2.260636451550656 /loss/FBt4/detection_loss‚2.8007581404277255 &loss/total_loss‚44.038869239338226 1loss/SRqqt4/detection_loss‚1.7628042426025659 /loss/FBt2/detection_loss‚0.6505803140675331 epoch‚4 0loss/SRqqt3/detection_loss‚1.895209188504917 /loss/FBt1/detection_loss‚3.0575753649075827 0loss/FRt2/assignment_loss‚1.3421528281437025 /loss/FRt3/assignment_loss‚3.397152697756177 .loss/FRt2/detection_loss‚1.392825746960524 1loss/SRqqt2/detection_loss‚0.7988170085350672 0loss/FRt4/assignment_loss‚3.4219309639930726 2loss/SRqqt2/assignment_loss‚0.9771864082788427 2loss/SRqqt3/assignment_loss‚1.7115384201269324 2loss/SRqqt4/assignment_loss‚1.6428931460699492 1loss/FBt3/assignment_loss‚0.29786338294707093 1loss/FBt1/assignment_loss‚0.19198250432799568 /loss/FRt1/detection_loss‚2.3662732814749083 .loss/FBt3/detection_loss‚2.774615615606308 % _timestamp‚1.7884221879238234e+09 _runtime‚5658.673665871bRP`’Ú .loss/FRt3/detection_loss‚2.367397218233063 0loss/SRqqt1/detection_loss‚2.282079409393999 _step‚483 /loss/FRt4/detection_loss‚2.2870238399505642 trainer/global_step‚5899 1loss/SRqqt1/assignment_loss‚2.260636451550656 /loss/FBt4/detection_loss‚2.8007581404277255 &loss/total_loss‚44.038869239338226 1loss/SRqqt4/detection_loss‚1.7628042426025659 /loss/FBt2/detection_loss‚0.6505803140675331 epoch‚4 0loss/SRqqt3/detection_loss‚1.895209188504917 /loss/FBt1/detection_loss‚3.0575753649075827 0loss/FRt2/assignment_loss‚1.3421528281437025 /loss/FRt3/assignment_loss‚3.397152697756177 .loss/FRt2/detection_loss‚1.392825746960524 1loss/SRqqt2/detection_loss‚0.7988170085350672 0loss/FRt4/assignment_loss‚3.4219309639930726 2loss/SRqqt2/assignment_loss‚0.9771864082788427 2loss/SRqqt3/assignment_loss‚1.7115384201269324 2loss/SRqqt4/assignment_loss‚1.6428931460699492 1loss/FBt3/assignment_loss‚0.29786338294707093 1loss/FBt1/assignment_loss‚0.19198250432799568 /loss/FRt1/detection_loss‚2.3662732814749083 .loss/FBt3/detection_loss‚2.774615615606308 % _timestamp‚1.7884221879238234e+09 _runtime‚5658.673665871 /loss/FRt1/assignment_loss‚3.989245746816908 1loss/FBt2/assignment_loss‚0.10427682638186869 0loss/FBt4/assignment_loss‚0.2660504922802959ãfæo’“‚(:… ·ÐäÔ¾‚Ÿx gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘Ë{”‚(:ð ·ÐäÔ{ disk.dm-0.in‚ 4.98046875 disk.dm-1.in‚7.3125& disk.nvme0c0n1.out‚25337.814453125 disk.sda.out‚ 304.02734375# memory_percent‚97.9527473449707 proc.cpu.threads‚17! disk.dm-0.out‚25337.814453125! disk.rbd2.out‚183.75927734375 cpu‚25.25986256507798 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.98046875 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125$ disk.nvme0n1.out‚25337.814453125) disk./.usagePercent‚73.64331316103193% disk./.usageGB‚1097.3385848999023 disk.rbd0.out‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 335.421875 network.sent‚ 1.3858832e+07 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 303.98828125# proc.memory.rssMB‚ 2862.76171875 disk.dm-1.out‚ 303.98828125 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 2.82275390625 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0n1.in‚ 4.98046875) proc.memory.percent‚17.47291088104248 network.recv‚ 1.991649e+06 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.rbd3.out‚09\ƒ)•‚(:… ÆÐäÔêß™x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yÉò·€–‚(:õ ÆÐäÔã¢óz disk.rbd1.out‚ 2.82275390625 disk.nvme0n1.in‚ 4.98046875 disk.dm-0.in‚ 4.98046875 disk.dm-1.out‚ 304.54296875 disk.rbd0.in‚0 disk.sda1.out‚0" memory_percent‚97.951340675354 disk.sda.out‚ 304.58203125 disk.dm-1.in‚7.3125! disk.rbd2.out‚185.33349609375 disk.rbd0.out‚0( proc.memory.availableMB‚ 335.65234375 network.recv‚ 1.997357e+06' disk./.usagePercent‚73.643444492897 disk.sda3.in‚7.3125 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0& disk.nvme0n1.out‚25338.51416015625 disk.rbd4.in‚0( disk.nvme0c0n1.out‚25338.51416015625 disk.sda1.in‚ 0.05078125 cpu‚25.25931309035874 proc.cpu.threads‚17# disk.dm-0.out‚25338.51416015625 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.rbd3.in‚0 network.sent‚ 1.3967523e+07 disk.sda3.out‚ 304.54296875 disk.drbd1777.out‚0 disk.sda2.out‚ 0.0390625 disk.nvme0c0n1.in‚ 4.98046875 disk.rbd3.out‚0# proc.memory.rssMB‚ 2862.76171875) proc.memory.percent‚17.47291088104248% disk./.usageGB‚1097.3405418395996 disk.sda2.in‚0ø¡N›—‚(: ÕÐäÔ¡¬—x gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04–“íu˜‚(:ê ÕÐäÔØ™«z disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 305.0390625 disk.rbd3.in‚0 disk.sda3.out‚ 305.0390625# proc.memory.rssMB‚ 2862.76171875 network.sent‚ 1.3973995e+07$ disk./.usageGB‚1097.168300628662 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚ 4.98046875) proc.memory.percent‚17.47291088104248# disk.nvme0n1.out‚25340.59765625 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.sda3.in‚7.3125 cpu‚25.258770013195573! disk.rbd2.out‚185.34912109375) disk./.usagePercent‚73.63188524071099 disk.rbd1.in‚0 disk.rbd1.out‚ 2.82275390625 disk.sda.in‚ 7.36328125 disk.sda.out‚ 305.078125 disk.dm-0.out‚25340.59765625 disk.rbd2.in‚0% disk.nvme0c0n1.out‚25340.59765625" memory_percent‚97.951340675354 disk.nvme0n1.in‚ 4.98046875 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 335.65234375 proc.cpu.threads‚17 network.recv‚ 2.001954e+06 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.dm-0.in‚ 4.98046875 disk.sda2.in‚0£æö2™- _wandbruntime‚5703  _runtime‚57030Ãɚà ' lr-AdamW/pg1‚0.00048174192914131233 ' lr-AdamW/pg2‚0.00048174192914131233 trainer/global_step‚5949 % _timestamp‚1.7884222331797392e+09 _runtime‚5703.929703936 _step‚484ó Î›È ' lr-AdamW/pg1‚0.00048174192914131233 ' lr-AdamW/pg2‚0.00048174192914131233 trainer/global_step‚5949 % _timestamp‚1.7884222331797392e+09 _runtime‚5703.929703936 _step‚484ä?]Ì2œ- _wandbruntime‚5703  _runtime‚5703Ä…†rí'çO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154534165 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154534165 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154508495 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11001934755360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11034171040768 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚263110296366080 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚233252505649696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚222250570894336 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263110597516288 trainer/global_step‚5949 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234715311256576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11002158404508 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 _runtime‚5703.930596087 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11227810114560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11034171040768 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109204075 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚245749236090880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251882787401728 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚234715311256576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚233252746338204 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚222250587933696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234715293253632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39835126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39834446 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11033942837248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11034171040768 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109204075 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚245749236090880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251882787401728 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚234715311256576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚233252746338204 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚222250587933696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234715293253632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39835126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39834446 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11033942837248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234715311256576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0å{;äù2Ÿ- _wandbruntime‚5704  _runtime‚5704Ç¥éÄö& ðM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234753631197696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚263153337486336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263755840 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154534046 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39840636 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11229404328448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚5704.803245674 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 trainer/global_step‚5949 $ _timestamp‚1.788422234052823e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103734086 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39841281 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109221796 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234753631197696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚234753649200640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109221796 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263781538 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154534046 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103733439 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263153634428928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚245789411729408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251923957097984 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263755840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222286873352192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚251923933157888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222286890391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11035780531712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11036008748544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263781538 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚233290640507152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245789657949184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11229677330944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11036008748544 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚233290881195868 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11003767154960 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154559742 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109221794 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109221794 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103733439 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263153634428928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚245789411729408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251923957097984 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263755840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222286873352192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚251923933157888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222286890391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11035780531712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11036008748544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263781538 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚233290640507152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245789657949184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11229677330944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11036008748544 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚233290881195868 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11003767154960 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154559742 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109221794 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109221794 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154534046 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39840636 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11229404328448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚5704.803245674 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 trainer/global_step‚5949 $ _timestamp‚1.788422234052823e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103734086 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39841281 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109221796 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234753631197696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚234753649200640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109221796 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696æ?­õ2¢- _wandbruntime‚5704  _runtime‚5704%-X£Ò .loss/FBt1/detection_loss‚2.922208309173584 % _timestamp‚1.7884222340544941e+09 2loss/SRqqt2/assignment_loss‚1.1811304020079514 1loss/FBt2/assignment_loss‚0.15180347871262417 0loss/FBt4/assignment_loss‚0.4568080274257227 %loss/total_loss‚45.09295802718027 /loss/FRt3/assignment_loss‚3.482814005613327 .loss/FRt4/detection_loss‚2.256170842796564 .loss/FRt1/detection_loss‚2.577529657718747 /loss/FRt2/detection_loss‚1.7689909863731135 1loss/SRqqt3/detection_loss‚2.0384540110826492 trainer/global_step‚5949 /loss/FRt2/assignment_loss‚2.051710421624391 2loss/SRqqt4/assignment_loss‚1.5675639293220793 0loss/FBt1/assignment_loss‚0.7691294577671215 /loss/FRt3/detection_loss‚2.4833491152524947 0loss/SRqqt1/detection_loss‚2.659898219837083 _step‚487 2loss/SRqqt1/assignment_loss‚2.1562908672624164 /loss/FBt2/detection_loss‚0.7236781876894053 /loss/FBt3/detection_loss‚2.8234242465760975 .loss/FBt4/detection_loss‚2.047165624797344 epoch‚4 /loss/FRt4/assignment_loss‚3.123906448483467 2loss/SRqqt3/assignment_loss‚1.5311773803247057 1loss/SRqqt2/detection_loss‚0.9605106923188413 _runtime‚5704.804321221 /loss/FRt1/assignment_loss‚3.434756595034932 1loss/FBt3/assignment_loss‚0.08346273916605343 1loss/SRqqt4/detection_loss‚1.8410243808195508µ„X­]¤× .loss/FRt4/detection_loss‚2.256170842796564 .loss/FRt1/detection_loss‚2.577529657718747 /loss/FRt2/detection_loss‚1.7689909863731135 1loss/SRqqt3/detection_loss‚2.0384540110826492 trainer/global_step‚5949 /loss/FRt2/assignment_loss‚2.051710421624391 2loss/SRqqt4/assignment_loss‚1.5675639293220793 0loss/FBt1/assignment_loss‚0.7691294577671215 /loss/FRt3/detection_loss‚2.4833491152524947 0loss/SRqqt1/detection_loss‚2.659898219837083 _step‚487 2loss/SRqqt1/assignment_loss‚2.1562908672624164 /loss/FBt2/detection_loss‚0.7236781876894053 /loss/FBt3/detection_loss‚2.8234242465760975 .loss/FBt4/detection_loss‚2.047165624797344 epoch‚4 /loss/FRt4/assignment_loss‚3.123906448483467 2loss/SRqqt3/assignment_loss‚1.5311773803247057 1loss/SRqqt2/detection_loss‚0.9605106923188413 _runtime‚5704.804321221 /loss/FRt1/assignment_loss‚3.434756595034932 1loss/FBt3/assignment_loss‚0.08346273916605343 1loss/SRqqt4/detection_loss‚1.8410243808195508 .loss/FBt1/detection_loss‚2.922208309173584 % _timestamp‚1.7884222340544941e+09 2loss/SRqqt2/assignment_loss‚1.1811304020079514 1loss/FBt2/assignment_loss‚0.15180347871262417 0loss/FBt4/assignment_loss‚0.4568080274257227 %loss/total_loss‚45.09295802718027 /loss/FRt3/assignment_loss‚3.482814005613327çÁÐ'¥‚(:„ äÐäÔ‘üœx gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.891' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1782 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇßÛ¿p¦‚(:å äÐäÔ³ñã{$ memory_percent‚97.95150756835938 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.rbd1.out‚ 2.82275390625 disk.dm-1.in‚7.3125 disk.nvme0n1.in‚ 4.98046875$ disk.nvme0n1.out‚25344.365234375 disk.sda2.in‚0 disk.dm-1.out‚ 305.71875 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.98046875 network.sent‚ 1.3979333e+07# proc.memory.availableMB‚335.625# proc.memory.rssMB‚ 2862.76171875 cpu‚25.258211436990447 disk.rbd3.in‚0 network.recv‚ 2.005371e+06) disk./.usagePercent‚73.63209337483632 disk.rbd2.in‚0 disk.rbd2.out‚ 185.447265625 disk.sda1.in‚ 0.05078125$ disk./.usageGB‚1097.171401977539 disk.sda.out‚ 305.7578125 disk.rbd1.in‚0 disk.dm-0.in‚ 4.98046875& disk.nvme0c0n1.out‚25344.365234375) proc.memory.percent‚17.47291088104248 disk.rbd3.out‚0 disk.sda3.out‚ 305.71875 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 proc.cpu.threads‚17! disk.dm-0.out‚25344.365234375 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.sda1.out‚0矧‚(:… óÐäÔ–í”x gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹¤ƒ¨‚(:ø óÐäÔçÎÔz disk.rbd4.in‚0 disk.sda3.in‚7.3125 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd3.in‚0 disk.sda.out‚ 306.05859375 network.recv‚ 2.011929e+06& disk.nvme0n1.out‚25611.53369140625# disk.dm-0.out‚25611.53369140625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 7.36328125# proc.memory.rssMB‚ 2862.76171875 disk.rbd1.in‚0 disk.dm-1.out‚ 306.01953125 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚25611.53369140625) proc.memory.percent‚17.47291088104248% disk./.usageGB‚1097.4315490722656 disk.dm-1.in‚7.3125 disk.rbd2.out‚ 185.462890625 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125( proc.memory.availableMB‚ 335.62890625 proc.cpu.threads‚17) disk./.usagePercent‚73.64955206464137 disk.nvme0n1.in‚ 4.98046875 disk.sda1.out‚0 disk.sda3.out‚ 306.01953125 cpu‚25.257679044417824 disk.rbd1.out‚ 2.82275390625 disk.dm-0.in‚ 4.98046875 disk.nvme0c0n1.in‚ 4.98046875 network.sent‚ 1.4088083e+07 disk.rbd4.out‚0 disk.rbd3.out‚0$ memory_percent‚97.95148372650146%q+º©‚(:… ‚ÑäÔàƒ¹x gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šþðyª‚(:î ‚ÑäÔéÇ—{) disk./.usagePercent‚73.64956844912356 disk.sda2.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 306.640625 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda3.out‚ 306.6015625# memory_percent‚97.9515552520752# proc.memory.rssMB‚ 2862.76171875% disk./.usageGB‚1097.4317932128906 disk.nvme0c0n1.in‚ 4.98046875' disk.nvme0c0n1.out‚25612.0244140625 disk.sda1.out‚0 disk.sda3.in‚7.3125 network.recv‚ 2.015779e+06" disk.dm-0.out‚25612.0244140625 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚ 4.98046875 network.sent‚ 1.4093898e+07 disk.dm-1.out‚ 306.6015625 disk.rbd0.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚25612.0244140625 disk.dm-0.in‚ 4.98046875 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚ 185.541015625 disk.rbd1.out‚ 2.82275390625) proc.memory.percent‚17.47291088104248 disk.sda2.out‚ 0.0390625' proc.memory.availableMB‚ 335.6171875 cpu‚25.25717787419414³rb2«- _wandbruntime‚5750  _runtime‚5750Œ¨×ǬÁ trainer/global_step‚5999 % _timestamp‚1.7884222798037724e+09 _runtime‚5750.553722184 _step‚488 & lr-AdamW/pg1‚0.0004760605791610504 & lr-AdamW/pg2‚0.0004760605791610504£•MÌ­Æ trainer/global_step‚5999 % _timestamp‚1.7884222798037724e+09 _runtime‚5750.553722184 _step‚488 & lr-AdamW/pg1‚0.0004760605791610504 & lr-AdamW/pg2‚0.0004760605791610504èŽ*Ù½2®- _wandbruntime‚5750  _runtime‚5750.¦ç„#¯çO `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11321025326080 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11126070177792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YLDevicÎà!µÊ&eStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 _step‚489 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64400200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64400198 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚236632190456832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11126070177792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚247758278637568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚247758032431104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110090025 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚5750.554888402 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104557391 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235159262113704 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110090027 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40157834 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚236632190456832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110090027 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚247758032431104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚224065693787136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚224065710826496 trainer/global_step‚5999 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112Ä,QÌò'°ìO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚236632190456832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11126070177792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚247758278637568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚247758032431104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110090025 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚5750.554888402 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104557391 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235159262113704 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110090027 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40157834 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚236632190456832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110090027 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚247758032431104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚224065693787136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚224065710826496 trainer/global_step‚5999 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11321025326080 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11126070177792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 _step‚489 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64400200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64400198 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0éÆQ‹Œ2±- _wandbruntime‚5751  _runtime‚5751êË N÷&²ñM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265948390 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚253982441910784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 _step‚490 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110107746 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155840644 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104573973 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11127907885568 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11095624375524 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11127907885568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235197637659876 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110107746 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155814948 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚247798454289408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224102013284352 _runtime‚5751.468707307 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚236670546403840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚247798208069632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11323167287296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235197396971160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265948390 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11127679668736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40163420 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265922692 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64410553 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚5999 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104574628 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64410555 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚236670528400896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11322894284800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11127679668736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚265305312255488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚247798454289408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚236670546403840 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40164073 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 % _timestamp‚1.7884222807181184e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561ðIÓÇø ³öM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155814948 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚247798454289408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224102013284352 _runtime‚5751.468707307 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚236670546403840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚247798208069632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11323167287296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235197396971160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265948390 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11127679668736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsk”.Monitor.on_train_batch_end/inactive_split.small_pool.freed‚40163420 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265922692 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64410553 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚5999 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104574628 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64410555 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚236670528400896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11322894284800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11127679668736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚265305312255488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 307470848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚247798454289408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚236670546403840 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40164073 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 % _timestamp‚1.7884222807181184e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265948390 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚253982441910784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 _step‚490 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110107746 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155840644 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104573973 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11127907885568 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11095624375524 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11127907885568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235197637659876 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110107746 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299êZÈ S2´- _wandbruntime‚5751  _runtime‚5751W}4YµÓ 0loss/FRt2/assignment_loss‚1.4641621234465618 /loss/FBt4/detection_loss‚2.6219194903969765 1loss/SRqqt1/assignment_loss‚2.025370235288782 2loss/SRqqt2/assignment_loss‚1.0977441587986854 /loss/FRt2/detection_loss‚1.8301010890578737 %loss/total_loss‚44.49180118259473 2loss/SRqqt3/assignment_loss‚1.7845179525748738 2loss/FBt1/assignment_loss‚0.043241113889962435 0loss/FBt3/assignment_loss‚0.0647702992913158 1loss/FBt4/assignment_loss‚0.15812810133502353 1loss/SRqqt1/detection_loss‚2.4267614228384837 1loss/SRqqt2/detection_loss‚0.9009486165757362 0loss/SRqqt3/detection_loss‚1.690652994052419 _runtime‚5751.470275562 /loss/FRt3/assignment_loss‚3.066203523427248 /loss/FRt1/detection_loss‚2.5471404066516294 .loss/FRt4/detection_loss‚2.205726315578243 /loss/FBt1/detection_loss‚3.3347309827804565 _step‚491 0loss/FBt2/assignment_loss‚0.1321383953794149 1loss/SRqqt4/detection_loss‚1.9604011601677127 /loss/FBt2/detection_loss‚0.6308775887016084 trainer/global_step‚5999 0loss/FRt4/assignment_loss‚3.5851111544503107 0loss/SRqqt4/assignment_loss‚1.58227080282043 .loss/FRt3/detection_loss‚2.400604007144769 /loss/FBt3/detection_loss‚2.7922544479370117 $ _timestamp‚1.788422280720405e+09 epoch‚4 /loss/FRt1/assignment_loss‚4.146024800009197S_á ^¶Ø /loss/FRt1/detection_loss‚2.5471404066516294 .loss/FRt4/detection_loss‚2.205726315578243 /loss/FBt1/detection_loss‚3.3347309827804565 _step‚491 0loss/FBt2/assignment_loss‚0.1321383953794149 1loss/SRqqt4/detection_loss‚1.9604011601677127 /loss/FBt2/detection_loss‚0.6308775887016084 trainer/global_step‚5999 0loss/FRt4/assignment_loss‚3.5851111544503107 0loss/SRqqt4/assignment_loss‚1.58227080282043 .loss/FRt3/detection_loss‚2.400604007144769 /loss/FBt3/detection_loss‚2.7922544479370117 $ _timestamp‚1.788422280720405e+09 epoch‚4 /loss/FRt1/assignment_loss‚4.146024800009197 0loss/FRt2/assignment_loss‚1.4641621234465618 /loss/FBt4/detection_loss‚2.6219194903969765 1loss/SRqqt1/assignment_loss‚2.025370235288782 2loss/SRqqt2/assignment_loss‚1.0977441587986854 /loss/FRt2/detection_loss‚1.8301010890578737 %loss/total_loss‚44.49180118259473 2loss/SRqqt3/assignment_loss‚1.7845179525748738 2loss/FBt1/assignment_loss‚0.043241113889962435 0loss/FBt3/assignment_loss‚0.0647702992913158 1loss/FBt4/assignment_loss‚0.15812810133502353 1loss/SRqqt1/detection_loss‚2.4267614228384837 1loss/SRqqt2/detection_loss‚0.9009486165757362 0loss/SRqqt3/detection_loss‚1.690652994052419 _runtime‚5751.470275562 /loss/FRt3/assignment_loss‚3.066203523427248ëOfêT·‚(:„ ‘ÑäÔɵx gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ᶸ‚(:ô ‘ÑäÔ”Ó¼{ disk.sda3.out‚ 307.0703125 disk.rbd3.in‚0 cpu‚25.256638111541257 proc.cpu.threads‚17 network.sent‚ 1.4100351e+07) disk./.usagePercent‚73.64955641676946% disk./.usageGB‚1097.4316139221191 disk.rbd1.out‚ 2.82275390625& disk.nvme0n1.out‚25612.98583984375 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 187.068359375 disk.nvme0c0n1.in‚ 4.98046875 disk.dm-1.in‚7.3125 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125$ memory_percent‚97.95553684234619# disk.dm-0.out‚25612.98583984375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚25612.98583984375( proc.memory.availableMB‚ 334.96484375) proc.memory.percent‚17.47291088104248 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125 disk.nvme0n1.in‚ 4.98046875 disk.rbd0.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2862.76171875 network.recv‚ 2.020583e+06 disk.dm-0.in‚ 4.98046875 disk.dm-1.out‚ 307.0703125 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 307.109375¹¬D€¹‚(:õ  ÑäÔƒ´| disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 2.82275390625 disk.dm-1.out‚ 307.56640625 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0# memory_percent‚97.9555606842041 disk.dm-1.in‚7.3125# proc.memory.rssMB‚ 2862.76171875 network.sent‚ 1.4208211e+07 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625( disk.nvme0c0n1.out‚25613.28271484375 disk.dm-0.in‚ 4.98046875' proc.memory.availableMB‚ 334.9609375 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda3.in‚7.3125) disk./.usagePercent‚73.64954131232493 disk.rbd2.out‚ 187.130859375 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 307.56640625 disk.nvme0n1.in‚ 4.98046875# disk.dm-0.out‚25613.28271484375 disk.nvme0c0n1.in‚ 4.98046875) proc.memory.percent‚17.47291088104248 network.recv‚ 2.026091e+06 disk.sda1.out‚0 disk.sda.out‚ 307.60546875 disk.rbd1.in‚0& disk.nvme0n1.out‚25613.28271484375 cpu‚25.25605597123622% disk./.usageGB‚1097.4313888549805 disk.drbd1777.in‚ 1.03515625(Ä”Žº‚(:ƒ  ÑäÔ—¬›ƒ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.03 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kŒÞ»‚(:… ¯ÑäÔÔÙ‚x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p«÷ƒ¼‚(:ø ¯ÑäÔþ׳{ disk.sda2.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 4.98046875 disk.sda3.out‚ 308.07421875 disk.nvme0n1.in‚ 4.98046875 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 308.07421875 disk.drbd1777.in‚ 1.03515625( proc.memory.availableMB‚ 335.20703125 cpu‚25.255530113207094 proc.cpu.threads‚17% disk./.usageGB‚1097.4397201538086& disk.nvme0n1.out‚25621.91748046875 disk.dm-0.in‚ 4.98046875 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚7.3125 disk.rbd0.out‚0# proc.memory.rssMB‚ 2862.76171875 network.sent‚ 1.4214239e+07 disk.rbd1.in‚0 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 308.11328125) disk./.usagePercent‚73.65010043277971 disk.rbd2.out‚ 187.146484375) proc.memory.percent‚17.47291088104248 disk.rbd1.out‚ 2.82275390625( disk.nvme0c0n1.out‚25621.91748046875$ memory_percent‚97.95405864715576 network.recv‚ 2.030676e+06# disk.dm-0.out‚25621.91748046875ã5ë•Ž½‚(:ƒ ¾ÑäÔƒþûw gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0";}zƒ¾‚(:ø ¾ÑäÔúä¢z disk.dm-0.in‚ 4.98046875# disk.dm-0.out‚25622.27294921875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.98046875 disk.rbd3.out‚0 disk.sda3.out‚ 308.77734375# proc.memory.rssMB‚ 2862.76171875) proc.memory.percent‚17.47291088104248 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125' proc.memory.availableMB‚ 334.9609375) disk./.usagePercent‚73.65010068878725 disk.nvme0n1.in‚ 4.98046875& disk.nvme0n1.out‚25622.27294921875 disk.dm-1.out‚ 308.77734375( disk.nvme0c0n1.out‚25622.27294921875# memory_percent‚97.9555606842041 network.sent‚ 1.4220112e+07 network.recv‚ 2.034536e+06% disk./.usageGB‚1097.4397239685059 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda.out‚ 308.81640625 cpu‚25.264228774695628 disk.dm-1.in‚7.3125 disk.rbd4.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0! disk.rbd2.out‚187.23681640625 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.rbd1.out‚ 2.82275390625 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0~m¿‚(:„ ÍÑäÔïšx gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎUåxƒÀ‚(:ø ÍÑäÔ§ÊÅz disk.rbd4.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 2.82275390625 disk.nvme0n1.in‚ 4.98046875 disk.sda2.in‚0) proc.memory.percent‚17.47291088104248 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 334.97265625 network.sent‚ 1.4225712e+07 disk.nvme0c0n1.in‚ 4.98046875 disk.sda3.in‚7.3125 disk.dm-0.in‚ 4.98046875 disk.sda2.out‚ 0.0390625! disk.rbd2.out‚187.25244140625 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda.out‚ 309.23046875 disk.rbd2.in‚0( disk.nvme0c0n1.out‚25622.82080078125 disk.rbd3.out‚0$ memory_percent‚97.95548915863037# proc.memory.rssMB‚ 2862.76171875 network.recv‚ 2.03867e+06& disk.nvme0n1.out‚25622.82080078125# disk.dm-0.out‚25622.82080078125 disk.dm-1.in‚7.3125) disk./.usagePercent‚73.65010222483245% disk./.usageGB‚1097.4397468566895 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.sda3.out‚ 309.19140625 disk.dm-1.out‚ 309.19140625 cpu‚25.27599180660293 disk.rbd0.in‚0 disk.rbd4.out‚0ÚA0:šÁ‚(: ÜÑäÔ³¨Éx gpu.0.gpu‚46.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.08' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.215999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á¢­½~‚(:ó ÜÑäÔ­õ¥{ disk.rbd4.in‚0& disk.nvme0n1.out‚25722.00439453125) disk./.usagePercent‚73.65847341519458 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.98046875$ memory_percent‚97.95553684234619 network.sent‚ 1.4230893e+07 disk.rbd3.in‚0( proc.memory.availableMB‚ 334.96484375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.98046875( disk.nvme0c0n1.out‚25722.00439453125# proc.memory.rssMB‚ 2862.76171875 network.recv‚ 2.041999e+06 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 309.7734375) proc.memory.percent‚17.47291088104248% disk./.usageGB‚1097.5644836425781 disk.sda.out‚309.8125 disk.sda3.in‚7.3125 disk.sda3.out‚ 309.7734375 cpu‚25.28705065898433 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.rbd1.out‚ 2.82275390625# disk.dm-0.out‚25722.00439453125 disk.dm-1.in‚7.3125! disk.rbd2.out‚187.37744140625 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.dm-0.in‚ 4.98046875 disk.sda2.in‚0 disk.rbd0.in‚0Âΰ2Ã- _wandbruntime‚5846  _runtime‚5846ï³FT*ÄÎT \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 549453824 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 269034581 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚250906338940928 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 541892608 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚268605124486144 ^QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚90898432 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 269014378 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚11229704358912 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚40712460 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31168 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 111636965 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚65315764 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20201 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚11229704358912 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚561 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 106028611 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚11426711642112 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 184860760 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31168 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚257178412844032 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 6893338624 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 189713920 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 269014378 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚299 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚262 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚11229876069888 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337968128 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚561 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 106028224 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 177287680 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 157377415 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6522528073 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚65315766 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚238145609361512 trainer/global_step‚6044 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 171710976 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚268604947198464 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6039636992 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 537294673 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20203 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 111636965 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚226948501361664 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚250906338940928 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 153347584 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚11229876069888 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚299 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 157397616 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6252512256 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1261 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚387 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚262 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1243 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚11426558294528 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 157377415 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6522528073 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚65315766 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚238145609361512 trainer/global_step‚6044 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 171710976 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚268604947198464 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6039636992 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 537294673 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20203 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 111636965 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚226948501361664 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚250906338940928 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 153347584 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚11229876069888 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚299 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 157397616 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6252512256 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1261 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚387 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚262 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1243 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚11426558294528 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31397 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3428 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20201 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 171710976 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚385 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚226973254310912 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚561 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20203 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 111651901 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 111651899 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6343884800 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 111651899 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚40717737 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 189713920 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚561 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚11230097154048 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3428 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 157405892 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 541892608 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31397 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚65324730 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 157385691 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6739344896 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚11230268865024 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚226973271350272 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚250932872525824 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚11230097154048 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1261 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 269057793 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚11427104437248 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31397 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3428 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20201 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 171710976 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚385 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚226973254310912 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚561 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20203 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 111651901 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 111651899 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6343884800 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 111651899 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚40717737 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 189713920 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚561 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚11230097154048 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3428 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 157405892 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 541892608 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31397 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚65324730 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 157385691 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6739344896 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚11230268865024 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚226973271350272 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚250932872525824 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚11230097154048 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1261 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 269057793 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚11427104437248 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 B&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4503366947174072 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9738703370094299 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17284245789051056 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.47566962242126465 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09567900747060776 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 +particle/specificity‚0.9477218389511108 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.¹yÁêù375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.504263162612915 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5170360803604126 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.005442176945507526 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.41885966062545776 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9892913103103638 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3819444477558136 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2180047482252121 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08810986578464508 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5301724672317505 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26458001136779785 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.40481704473495483 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0789031982421875 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17669755220413208 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.35044094920158386 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.37013888359069824 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21529492735862732 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 _step‚494 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31296294927597046 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07145210355520248 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3541666567325592 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1304609179496765 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5169652104377747 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13161177933216095 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5265912413597107 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 _runtime‚ 5861.00529093 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 particle/accuracy_6_of_7‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.33622244000434875 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004295366816222668 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_2‚0.0854254812002182 ,jet/accuracy_4_of_4‚0.002844839356839657 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1842660754919052 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.48629409074783325 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.45161548256874084 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_5‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24629631638526917 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 jet/accuracy_6_of_6‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.22499999403953552 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4917179048061371 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7480994462966919 ,jet/accuracy_3_of_3‚0.017167920246720314 +particle/sensitivity‚0.2859865725040436 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1327555924654007 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.49954459071159363 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02142857201397419 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8872072696685791 particle/accuracy_1_of_1‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.46402719616889954 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4266666769981384 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2866339087486267 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_1‚0.4788822829723358 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1700681447982788 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16758926212787628 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4746089279651642 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4689562916755676 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02427203394472599 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.46779415011405945 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.40666663646698 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5162955522537231 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4019526243209839 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 +jet/accuracy_2_of_6‚0.38461539149284363 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.988304078578949 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22836461663246155 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17180238664150238 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6526291966438293 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008818342466838658 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10222222656011581 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4107142984867096 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.46582165360450745 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 jet/accuracy_1_of_6‚1 particle/accuracy_4_of_6‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 % _timestamp‚1.7884223902534752e+09 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9785714745521545 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4148055613040924 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13822992146015167 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5244367718696594 (particle/accuracy‚0.8443804383277893 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_7_of_7‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48306578397750854 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 particle/accuracy_3_of_7‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4803158938884735 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49960628151893616 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4451952278614044 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.39578551054000854 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5601851940155029 +jet/accuracy_3_of_4‚0.05439027026295662 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 particle/accuracy_1_of_4‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.45858466625213623 7"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.51412433385849 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.09493862092494965 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 particle/accuracy_6_of_6‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25327596068382263 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 particle/accuracy_2_of_6‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13302560150623322 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.05319149047136307 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18565747141838074 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14437688887119293 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4132680296897888 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2669062316417694 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0635833665728569 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.25317710638046265 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12646622955799103 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12222222238779068 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 )jet/accuracy_1_of_3‚0.857126772403717 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 , Purity/0FRt0SRqqt3FBt/FBt_purity‚0.90625 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_7‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012196033261716366 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5624999403953552 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.29440367221832275 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.497884064912796 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.812038779258728 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4375000298023224 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23393075168132782 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.11915584653615952 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1410919725894928 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 jet/accuracy_2_of_7‚0 particle/accuracy_5_of_7‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.056570302695035934 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4881020784378052 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5178571939468384 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3948343098163605 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 particle/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9089835286140442 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22145427763462067 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.021966207772493362 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4193996787071228 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13323666155338287 ,jet/accuracy_4_of_5‚0.010080644860863686 particle/accuracy_3_of_5‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11743941903114319 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 jet/accuracy_3_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666865348816 particle/accuracy_3_of_6‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6316826343536377 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5247023701667786 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.15266717970371246 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.44117653369903564 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5123241543769836 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4351870119571686 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.07067332416772842 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09637384116649628 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3270060420036316 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04729168489575386 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01595892570912838 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1380671262741089 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.16717171669006348 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5263158082962036 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.504263162612915 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.48306381702423096 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9873574376106262 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.30223217606544495 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9818478226661682 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9683907628059387 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9683627486228943 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9854815602302551 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9428820013999939 *jet/accuracy_2_of_3‚0.2085932195186615 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9916114211082458 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1128929927945137 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4335317611694336 jet/accuracy_7_of_7‚0 particle/accuracy_4_of_7‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_7‚0.5 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.022969914600253105 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_1_of_2‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.011185983195900917 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.38081395626068115 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1429673731327057 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992441534996033 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.44466233253479004 jet/accuracy_4_of_7‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 epoch‚4 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9858606457710266 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4837963283061981 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1760101169347763 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35228171944618225 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.453235924243927 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.38282671570777893 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11612974107265472 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9995822310447693 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5273877382278442 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.31718751788139343 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.13265305757522583 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13822992146015167 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13085272908210754 particle/accuracy_3_of_3‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.49076569080352783 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33036744594573975 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.48923608660697937 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9777105450630188 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 jet/accuracy_5_of_5‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9837728142738342 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9853567481040955 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9778362512588501 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.14684684574604034 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 jet/accuracy_5_of_6‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1839454472064972 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48923152685165405 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.986941933631897 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.984291672706604 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.013888888992369175 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9882943034172058 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4687366783618927 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 6"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5112344622612 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5049552321434021 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.30949151515960693 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.010507569648325443 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23082759976387024 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.41708871722221375 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.491170734167099 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5624999403953552 )loss/val_total_loss‚43.84143795709338 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12557362020015717 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.0909961685538292 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2857142984867096 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.983612060546875 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4870222508907318 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.012320484034717083 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1728164106607437 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4168103337287903 particle/accuracy_2_of_5‚1 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970349073410034 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*F‚ %9Rt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5150531530380249 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9960294365882874 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14729218184947968 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21149425208568573 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3219697177410126 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4879908263683319 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 *jet/accuracy_1_of_5‚0.9490463733673096 6validation_average_jet_accuracy‚0.4235221743583679 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5055564045906067 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22231024503707886 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_2_of_2‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9881570935249329 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.46834009885787964 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.49249082803726196 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 trainer/global_step‚6044 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1611262857913971 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1914682388305664 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5089726448059082 particle/accuracy_2_of_4‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992441534996033 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9906619191169739 &particle/f_score‚0.364487886428833 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0714285746216774 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9947217702865601 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.46893981099128723 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05633305385708809 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14321433007717133 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.03846153989434242 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4526045322418213 ("Purity/3FRt1SRqqt2FBt/event_purity‚0ÛÏ1 ¹qÍŽ¦ *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_5‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24629631638526917 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 jet/accuracy_6_of_6‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.22499999403953552 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4917179048061371 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7480994462966919 ,jet/accuracy_3_of_3‚0.017167920246720314 +particle/sensitivity‚0.2859865725040436 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1327555924654007 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.49954459071159363 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02142857201397419 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8872072696685791 particle/accuracy_1_of_1‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.46402719616889954 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4266666769981384 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2866339087486267 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_1‚0.4788822829723358 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1700681447982788 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16758926212787628 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4746089279651642 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4689562916755676 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02427203394472599 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.46779415011405945 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.40666663646698 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5162955522537231 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4019526243209839 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 +jet/accuracy_2_of_6‚0.38461539149284363 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.988304078578949 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22836461663246155 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17180238664150238 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6526291966438293 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008818342466838658 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10222222656011581 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4107142984867096 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.46582165360450745 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 jet/accuracy_1_of_6‚1 particle/accuracy_4_of_6‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 % _timestamp‚1.7884223902534752e+09 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9785714745521545 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4148055613040924 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13822992146015167 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5244367718696594 (particle/accuracy‚0.8443804383277893 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_7_of_7‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48306578397750854 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 particle/accuracy_3_of_7‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4803158938884735 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49960628151893616 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4451952278614044 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.39578551054000854 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5601851940155029 +jet/accuracy_3_of_4‚0.05439027026295662 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 particle/accuracy_1_of_4‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.45858466625213623 7"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.51412433385849 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.09493862092494965 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 particle/accuracy_6_of_6‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25327596068382263 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 particle/accuracy_2_of_6‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13302560150623322 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.05319149047136307 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18565747141838074 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14437688887119293 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4132680296897888 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2669062316417694 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0635833665728569 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.25317710638046265 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12646622955799103 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12222222238779068 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 )jet/accuracy_1_of_3‚0.857126772403717 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 , Purity/0FRt0SRqqt3FBt/FBt_purity‚0.90625 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_7‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012196033261716366 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5624999403953552 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.29440367221832275 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.497884064912796 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.812038779258728 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4375000298023224 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23393075168132782 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.11915584653615952 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1410919725894928 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 jet/accuracy_2_of_7‚0 particle/accuracy_5_of_7‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.056570302695035934 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4881020784378052 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5178571939468384 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3948343098163605 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 particle/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9089835286140442 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22145427763462067 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.021966207772493362 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4193996787071228 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13323666155338287 ,jet/accuracy_4_of_5‚0.010080644860863686 particle/accuracy_3_of_5‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11743941903114319 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 jet/accuracy_3_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666865348816 particle/accuracy_3_of_6‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6316826343536377 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5247023701667786 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.15266717970371246 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.44117653369903564 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5123241543769836 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4351870119571686 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.07067332416772842 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09637384116649628 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3270060420036316 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04729168489575386 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01595892570912838 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1380671262741089 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.16717171669006348 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5263158082962036 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.504263162612915 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.48306381702423096 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9873574376106262 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.30223217606544495 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9818478226661682 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9683907628059387 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9683627486228943 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9854815602302551 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9428820013999939 *jet/accuracy_2_of_3‚0.2085932195186615 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9916114211082458 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1128929927945137 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4335317611694336 jet/accuracy_7_of_7‚0 particle/accuracy_4_of_7‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_7‚0.5 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.022969914600253105 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_1_of_2‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.011185983195900917 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.38081395626068115 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1429673731327057 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992441534996033 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.44466233253479004 jet/accuracy_4_of_7‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 epoch‚4 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9858606457710266 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4837963283061981 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1760101169347763 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35228171944618225 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.453235924243927 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.38282671570777893 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11612974107265472 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9995822310447693 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5273877382278442 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.31718751788139343 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.13265305757522583 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13822992146015167 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13085272908210754 particle/accuracy_3_of_3‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.49076569080352783 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33036744594573975 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.48923608660697937 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9777105450630188 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 jet/accuracy_5_of_5‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9837728142738342 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9853567481040955 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9778362512588501 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.14684684574604034 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 jet/accuracy_5_of_6‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1839454472064972 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48923152685165405 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.986941933631897 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.984291672706604 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.013888888992369175 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9882943034172058 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4687366783618927 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 6"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5112344622612 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5049552321434021 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.30949151515960693 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.010507569648325443 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23082759976387024 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.41708871722221375 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.491170734167099 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5624999403953552 )loss/val_total_loss‚43.84143795709338 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12557362020015717 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.0909961685538292 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2857142984867096 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.983612060546875 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4870222508907318 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.012320484034717083 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1728164106607437 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4168103337287903 particle/accuracy_2_of_5‚1 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970349073410034 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5150531530380249 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9960294365882874 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14729218184947968 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21149425208568573 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3219697177410126 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_puriY6z\!ty‚0.4879908263683319 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 *jet/accuracy_1_of_5‚0.9490463733673096 6validation_average_jet_accuracy‚0.4235221743583679 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5055564045906067 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22231024503707886 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_2_of_2‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9881570935249329 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.46834009885787964 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.49249082803726196 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 trainer/global_step‚6044 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1611262857913971 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1914682388305664 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5089726448059082 particle/accuracy_2_of_4‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992441534996033 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9906619191169739 &particle/f_score‚0.364487886428833 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0714285746216774 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9947217702865601 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.46893981099128723 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05633305385708809 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14321433007717133 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.03846153989434242 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4526045322418213 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 +jet/accuracy_2_of_4‚0.30968666076660156 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01988079398870468 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03017241507768631 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4503366947174072 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9738703370094299 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17284245789051056 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.47566962242126465 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09567900747060776 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 +particle/specificity‚0.9477218389511108 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.504263162612915 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5170360803604126 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.005442176945507526 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.41885966062545776 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9892913103103638 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3819444477558136 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2180047482252121 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08810986578464508 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5301724672317505 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26458001136779785 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.40481704473495483 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0789031982421875 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17669755220413208 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.35044094920158386 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.37013888359069824 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21529492735862732 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 _step‚494 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31296294927597046 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07145210355520248 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3541666567325592 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1304609179496765 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5169652104377747 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13161177933216095 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5265912413597107 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 _runtime‚ 5861.00529093 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 particle/accuracy_6_of_7‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.33622244000434875 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004295366816222668 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_2‚0.0854254812002182 ,jet/accuracy_4_of_4‚0.002844839356839657 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1842660754919052 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.48629409074783325 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.45161548256874084 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0îdá4.›Î‚(: úÑäÔ±èÀx gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.115' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.622999999999998 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôùvÏ‚(:ë úÑäÔþ׫{ disk.rbd3.out‚0 disk.rbd1.out‚ 2.82275390625& disk.nvme0n1.out‚25889.77880859375 disk.sda2.out‚ 0.0390625 disk.sda1.in‚ 0.05078125! proc.memory.availableMB‚507.5# proc.memory.rssMB‚ 2861.77734375) disk./.usagePercent‚73.66697696145191 disk.sda3.in‚7.3125 disk.rbd4.in‚0( disk.nvme0c0n1.out‚25889.77880859375 cpu‚25.305829571683446 proc.cpu.threads‚17 network.recv‚ 2.052553e+06 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 310.8203125% disk./.usageGB‚1097.6911926269531 disk.sda3.out‚ 310.78125 disk.dm-0.in‚ 4.98046875 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 310.78125 disk.rbd2.in‚0# memory_percent‚96.9024658203125 network.sent‚ 1.4345286e+07 disk.sda2.in‚0! disk.rbd2.out‚189.02587890625 disk.nvme0c0n1.in‚ 4.98046875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 4.98046875 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.sda.in‚ 7.36328125) proc.memory.percent‚17.46690273284912# disk.dm-0.out‚25889.77880859375 disk.rbd0.in‚0 disk.rbd4.out‚0T`$à2Ð- _wandbruntime‚5868  _runtime‚5868Xp¨ÇÑÁ & lr-AdamW/pg1‚0.0004703636198738802 & lr-AdamW/pg2‚0.0004703636198738802 trainer/global_step‚6049 % _timestamp‚1.7884223974683595e+09 _runtime‚5868.218318079 _step‚495¸ðñÌÒÆ % _timestamp‚1.7884223974683595e+09 _runtime‚5868.218318079 _step‚495 & lr-AdamW/pg1‚0.0004703636198738802 & lr-AdamW/pg2‚0.0004703636198738802 trainer/global_step‚6049ï ½ª2Ó- _wandbruntime‚5868  _runtime‚5868d2WÐì'ÔæO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269510567 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚251438461610496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257726877575168 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157619294 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚238650115632162 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106294971 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257726901515264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚6049 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269536239 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11242732399104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269166392037376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚227440252482560 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚251438707816960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40811680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚238650356320670 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240195957414912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106294360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111916943 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11210086798750 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65482682 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11209863149602 _runtime‚5868.219652073 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11242504195584 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111916945 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157593624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚251438707816960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65482680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚227440269521920 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚251438461610496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11439514462208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 $ _timestamp‚1.788422397469398e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111916943 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111916945 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157593624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚251438707816960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65482680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚227440269521920 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚251438461610496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11439514462208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 $ _timestamp‚1.788422397469398e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111916943 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚251438461610496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257726877575168 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157619294 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚238650115632162 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106294971 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257726901515264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚6049 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269536239 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11242732399104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269166392037376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚227440252482560 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚251438707816960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40811680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚238650356320670 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240195957414912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106294360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111916943 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11210086798750 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65482682 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11209863149602 _runtime‚5868.219652073 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11242504195584 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238688491178334 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269553837 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257768047271424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227476571979776 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111934662 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227476554940416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ;4DeviceStatsMonitor.on_train_{¦C· batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚240234295358976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11441386050048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11244570106880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚240234313361920 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240234295358976 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106311900 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111934662 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157619175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚251478883468800 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269579535 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65493035 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11244341890048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚251478637249024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11244341890048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11244570106880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157644871 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269209732361216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚238688250489618 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269579535 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111934664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40818863 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157619175 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257768071211520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚251478883468800 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106311276 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624øxCü&ØöM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚240234295358976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11441386050048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11244570106880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚240234313361920 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240234295358976 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106311900 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111934662 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157619175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚251478883468800 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269579535 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65493035 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11244341890048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚251478637249024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11244341890048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11244570106880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157644871 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269209732361216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚238688250489618 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269579535 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111934664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40818863 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157619175 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257768071211520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚251478883468800 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106311276 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238688491178334 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269553837 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257768047271424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227476571979776 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111934662 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227476554940416ñ‰)ûÜ2Ù- _wandbruntime‚5869  _runtime‚5869N8¼‚^ÚØ &loss/total_loss‚43.798292921886315 % _timestamp‚1.7884223983552492e+09 0loss/FRt2/assignment_loss‚1.7538934653660037 0loss/FRt4/assignment_loss‚3.1020081654191016 1loss/FBt2/assignment_loss‚0.15142275900136548 1loss/SRqqt4/detection_loss‚2.0677518497924416 _step‚498 1loss/FBt4/assignment_loss‚0.22094628126360477 epoch‚5 1loss/SRqqt3/detection_loss‚1.8881164421992642 .loss/FBt1/detection_loss‚3.202642480532328 2loss/SRqqt1/assignment_loss‚1.8410611925301728 2loss/SRqqt2/assignment_loss‚0.9504776143003255 /loss/FRt1/detection_loss‚2.7923913864475307 1loss/SRqqt1/detection_loss‚2.5704444735138505 /loss/FBt2/detection_loss‚0.5289358454139297 _runtime‚ 5869.10508379 /loss/FRt3/detection_loss‚2.3284974667158993 /loss/FBt3/detection_loss‚2.4364250749349594 /loss/FBt4/detection_loss‚2.2453000426292418 0loss/FBt1/assignment_loss‚0.4864063701437165 /loss/FRt2/detection_loss‚1.6683159525383193 /loss/FRt4/detection_loss‚2.2889805808684534 1loss/SRqqt2/detection_loss‚0.7917189654544927 2loss/SRqqt3/assignment_loss‚1.5648457554302044 trainer/global_step‚6049 /loss/FRt1/assignment_loss‚3.885025421778361 /loss/FRt3/assignment_loss‚2.958781982010061 2loss/SRqqt4/assignment_loss‚1.3934194054834697 0loss/FBt3/assignment_loss‚0.6804839481192175sçBcÛÝ 0loss/FRt4/assignment_loss‚3.1020081654191016 1loss/FBt2/assignment_loss‚0.15142275900136548 1loss/SRqqt4/detection_loss‚2.0677518497924416 _step‚498 1loss/FBt4/assignment_loss‚0.22094628126360477 epoch‚5 1loss/SRqqt3/detection_loss‚1.8881164421992642 .loss/FBt1/detection_loss‚3.202642480532328 2loss/SRqqt1/assignment_loss‚1.8410611925301728 2loss/SRqqt2/assignment_loss‚0.9504776143003255 /loss/FRt1/detection_loss‚2.7923913864475307 1loss/SRqqt1/detection_loss‚2.5704444735138505 /loss/FBt2/detection_loss‚0.5289358454139297 _runtime‚ 5869.10508379 /loss/FRt3/detection_loss‚2.3284974667158993 /loss/FBt3/detection_loss‚2.4364250749349594 /loss/FBt4/detection_loss‚2.2453000426292418 0loss/FBt1/assignment_loss‚0.4864063701437165 /loss/FRt2/detection_loss‚1.6683159525383193 /loss/FRt4/detection_loss‚2.2889805808684534 1loss/SRqqt2/detection_loss‚0.7917189654544927 2loss/SRqqt3/assignment_loss‚1.5648457554302044 trainer/global_step‚6049 /loss/FRt1/assignment_loss‚3.885025421778361 /loss/FRt3/assignment_loss‚2.958781982010061 2loss/SRqqt4/assignment_loss‚1.3934194054834697 0loss/FBt3/assignment_loss‚0.6804839481192175 &loss/total_loss‚43.798292921886315 % _timestamp‚1.7884223983552492e+09 0loss/FRt2/assignment_loss‚1.7538934653660037òè|!’šÜ‚(: ‰ÒäÔ¦óšx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.08' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.215999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ÙHzÝ‚(:ï ‰ÒäÔªÑÐz disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 2.82275390625! disk.rbd2.out‚189.04150390625 disk.rbd4.out‚0 disk.sda1.out‚0 disk.sda.out‚ 311.3359375 network.sent‚ 1.4469481e+07 network.recv‚ 2.058867e+06 disk.dm-0.in‚ 4.98046875 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 4.98046875 disk.dm-1.out‚ 311.296875$ memory_percent‚96.90189361572266 cpu‚25.305143899387723" disk.dm-0.out‚25989.7685546875 disk.rbd2.in‚0' disk.nvme0c0n1.out‚25989.7685546875 disk.sda3.out‚ 311.296875 disk.rbd1.in‚0# proc.memory.rssMB‚ 2861.77734375 disk.sda2.out‚ 0.0390625 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125) proc.memory.percent‚17.46690273284912) disk./.usagePercent‚73.66695059267587% disk.nvme0n1.out‚25989.7685546875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.98046875 disk.sda1.in‚ 0.05078125% proc.memory.availableMB‚ 507.59375% disk./.usageGB‚1097.6907997131348 disk.dm-1.in‚7.3125•Ý³›Þ‚(: ˜ÒäÔ›é©x gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&0·xß‚(:í ˜ÒäÔƒàª{( proc.memory.availableMB‚ 507.34765625 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125" memory_percent‚96.903395652771% disk./.usageGB‚1097.9402389526367! disk.dm-0.out‚26255.919921875 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0& disk.nvme0c0n1.out‚26255.919921875 disk.sda1.out‚0 disk.rbd3.in‚0$ disk.nvme0n1.out‚26255.919921875 disk.sda3.in‚7.3125 network.sent‚ 1.4577093e+07 disk.dm-0.in‚ 4.98046875 disk.sda2.in‚0 disk.dm-1.out‚ 312.015625 disk.sda3.out‚ 312.015625 cpu‚25.304491346863593 disk.rbd3.out‚0# proc.memory.rssMB‚ 2861.77734375 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.98046875 disk.sda.in‚ 7.36328125) disk./.usagePercent‚73.68369066933231 disk.sda2.out‚ 0.0390625! disk.rbd2.out‚189.20263671875) proc.memory.percent‚17.46690273284912 proc.cpu.threads‚17 network.recv‚ 2.064121e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 2.82275390625 disk.nvme0n1.in‚ 4.98046875 disk.dm-1.in‚7.3125 disk.rbd4.out‚0 disk.sda.out‚ 312.0546875 )××à‚(:„ §ÒäԜ̲x gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷½ûœuá‚(:ê §ÒäÔÒòz disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 2.82275390625 disk.drbd1777.in‚ 1.03515625# proc.memory.rssMB‚ 2861.77734375 network.recv‚ 2.068843e+06 disk.sda.out‚ 312.421875 disk.sda3.out‚ 312.3828125 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0! disk.rbd2.out‚189.21826171875 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125% disk.nvme0c0n1.out‚26256.90234375) proc.memory.percent‚17.46690273284912 cpu‚25.30385225881095) disk./.usagePercent‚73.68369066933231 disk.rbd4.out‚0( proc.memory.availableMB‚ 507.34765625 disk.nvme0n1.in‚ 4.98046875 disk.rbd4.in‚0" memory_percent‚96.903395652771% disk./.usageGB‚1097.9402389526367 disk.rbd3.out‚0# disk.nvme0n1.out‚26256.90234375 disk.dm-0.in‚ 4.98046875 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.98046875 disk.rbd0.out‚0 network.sent‚ 1.4583169e+07 disk.rbd3.in‚0 disk.dm-0.out‚26256.90234375 disk.dm-1.out‚ 312.3828125 disk.sda1.out‚0 proc.cpu.threads‚17]5˜½2â- _wandbruntime‚5914  _runtime‚5914/à–¶ÇãÁ trainer/global_step‚6099 % _timestamp‚1.7884224436198487e+09 _runtime‚5914.369784257 _step‚499 & lr-AdamW/pg1‚0.0004646522385175783 & lr-AdamW/pg2‚0.0004646522385175783*F<ÌäÆ trainer/global_step‚6099 % _timestamp‚1.7884224436198487e+09 _runtime‚5914.369784257 _step‚499 & lr-AdamW/pg1‚0.0004646522385175783 & lr-AdamW/pg2‚0.0004646522385175783ó¦ôò§2å- _wandbruntime‚5914  _runtime‚5914ßNZÈææO trainer/global_step‚6099 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚253447257950720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229255392414720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253447504157184 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107134667 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11532929936384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11334631536128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112802893 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242112854618112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11334631536128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259785362387968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229255375375360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253447257950720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚242112854618112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242112872621056 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158900196 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 $ _timestamp‚1.788422443620918e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11301720369958 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259785386328064 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0x´Hñ'çëO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11301720369958 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259785386328064 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚6099 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚253447257950720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229255392414720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253447504157184 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107134667 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11532929936384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11334631536128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112802893 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242112854618112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11334631536128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259785362387968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229255375375360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253447257950720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚242112854618112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242112872621056 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158900196 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 $ _timestamp‚1.788422443620918e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673ôô ÞË2è- _wandbruntime‚5915  _runtime‚5915Û‚h¯÷&éñM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271746387 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112820612 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66010785 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242151210565120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41141111 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884224444933126e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112820612 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271746387 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107151258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242151192562176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚271361324761088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _runtime‚5915.244025691 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11336241027072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11336469243904 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158900077 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253487679809024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112820614 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41140473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242151192562176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271720689 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 _step‚501 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11336241027072 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚259826532084224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 trainer/global_step‚6099 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112820614 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚253487433589248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253487679809024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11336469243904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚271361625897984 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 271720689 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229291694872576 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107151898 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242151210565120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11303329120410 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158925773 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158925773 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66010787 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚240595006953626 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229291677833216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11535069873664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚240595247642342 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112820612 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66010785 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242151210565120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41141111 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884224444933126e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112820612 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271746387 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107151258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242151192562176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚271361324761088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _runtime‚5915.244025691 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11336241027072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11336469243904 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158900077 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253487679809024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112820614 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41140473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242151192562176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271720689 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 _step‚501 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11336241027072 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚259826532084224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 trainer/global_step‚6099 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112820614 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚253487433589248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253487679809024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11336469243904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚271361625897984 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 271720689 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229291694872576 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107151898 QBDeviceStatsMonitor. éù]on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242151210565120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11303329120410 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158925773 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158925773 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66010787 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚240595006953626 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229291677833216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11535069873664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚240595247642342õ€|G2ë- _wandbruntime‚5915  _runtime‚5915˜@1»^ìØ 0loss/SRqqt1/detection_loss‚2.547309979370662 1loss/SRqqt3/detection_loss‚1.9534635899419135 1loss/SRqqt4/detection_loss‚1.7969328128561681 _step‚502 0loss/FRt3/assignment_loss‚2.5779204607009887 1loss/FBt4/assignment_loss‚0.37678793537197636 _runtime‚5915.245148787 % _timestamp‚1.7884224444952028e+09 1loss/FBt3/assignment_loss‚0.40942917099561205 0loss/SRqqt2/detection_loss‚0.904165388903383 /loss/FBt2/detection_loss‚0.5827110504130056 0loss/FRt2/assignment_loss‚0.9160588497811175 2loss/SRqqt2/assignment_loss‚1.0161657622930678 .loss/FRt3/detection_loss‚2.514050120777554 .loss/FBt4/detection_loss‚3.634429156780243 .loss/FRt1/detection_loss‚2.274804443741838 /loss/FBt1/detection_loss‚2.5065559446811676 &loss/total_loss‚42.403773622225216 epoch‚5 trainer/global_step‚6099 /loss/FRt2/detection_loss‚1.6742135796775208 0loss/FRt1/assignment_loss‚3.4073839696745076 0loss/FRt4/assignment_loss‚3.5832139924168587 2loss/SRqqt3/assignment_loss‚1.2986201642250472 .loss/FBt3/detection_loss‚2.219361337748441 2loss/SRqqt4/assignment_loss‚1.6312326679424363 1loss/FBt1/assignment_loss‚0.23279185947030784 2loss/SRqqt1/assignment_loss‚1.9159849658608437 1loss/FBt2/assignment_loss‚0.08730522086761683 /loss/FRt4/detection_loss‚2.3428811977329427óé&ÇcíÝ 0loss/FRt2/assignment_loss‚0.9160588497811175 2loss/SRqqt2/assignment_loss‚1.0161657622930678 .loss/FRt3/detection_loss‚2.514050120777554 .loss/FBt4/detection_loss‚3.634429156780243 .loss/FRt1/detection_loss‚2.274804443741838 /loss/FBt1/detection_loss‚2.5065559446811676 &loss/total_loss‚42.403773622225216 epoch‚5 trainer/global_step‚6099 /loss/FRt2/detection_loss‚1.6742135796775208 0loss/FRt1/assignment_loss‚3.4073839696745076 0loss/FRt4/assignment_loss‚3.5832139924168587 2loss/SRqqt3/assignment_loss‚1.2986201642250472 .loss/FBt3/detection_loss‚2.219361337748441 2loss/SRqqt4/assignment_loss‚1.6312326679424363 1loss/FBt1/assignment_loss‚0.23279185947030784 2loss/SRqqt1/assignment_loss‚1.9159849658608437 1loss/FBt2/assignment_loss‚0.08730522086761683 /loss/FRt4/detection_loss‚2.3428811977329427 0loss/SRqqt1/detection_loss‚2.547309979370662 1loss/SRqqt3/detection_loss‚1.9534635899419135 1loss/SRqqt4/detection_loss‚1.7969328128561681 _step‚502 0loss/FRt3/assignment_loss‚2.5779204607009887 1loss/FBt4/assignment_loss‚0.37678793537197636 _runtime‚5915.245148787 % _timestamp‚1.7884224444952028e+09 1loss/FBt3/assignment_loss‚0.40942917099561205 0loss/SRqqt2/detection_loss‚0.904165388903383 /loss/FBt2/detection_loss‚0.5827110504130056öЇNïŽî‚(:ƒ ¶ÒäÔø¦™x gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßõÞ”…ï‚(:ú ¶ÒäÔ»ÃÑ{ disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 4.98046875 network.sent‚ 1.4589003e+07) disk./.usagePercent‚73.68368938929464 disk.sda.out‚ 313.04296875& disk.nvme0n1.out‚26356.11962890625 disk.dm-1.in‚7.3125$ memory_percent‚96.90859317779541 network.recv‚ 2.072717e+06 disk.rbd1.in‚0 disk.dm-1.out‚ 313.00390625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.303205957589174% disk./.usageGB‚1097.9402198791504 disk.drbd1777.in‚ 1.03515625# proc.memory.rssMB‚ 2861.77734375( disk.nvme0c0n1.out‚26356.11962890625 disk.rbd1.out‚ 2.82275390625 disk.nvme0n1.in‚ 4.98046875 disk.rbd4.out‚0( proc.memory.availableMB‚ 506.49609375 disk.rbd4.in‚0# disk.dm-0.out‚26356.11962890625 disk.sda.in‚ 7.36328125 disk.dm-0.in‚ 4.98046875 disk.sda1.out‚0 disk.rbd3.out‚0) proc.memory.percent‚17.46690273284912 disk.sda3.out‚ 313.00390625! disk.rbd2.out‚190.85498046875 proc.cpu.threads‚17 disk.sda3.in‚7.3125 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.sda1.in‚ 0.05078125—5ð‚(:„ ÅÒäÔØÈ•x gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þñ‚(:ö ÅÒäÔ•µÐz disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 2.90869140625( disk./.usagePercent‚73.6836988615734& disk.nvme0n1.out‚26455.39990234375# disk.dm-0.out‚26455.39990234375 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.sda3.in‚7.3125 cpu‚25.302536261136353 network.recv‚ 2.079382e+06 disk.nvme0n1.in‚ 4.98046875 disk.dm-0.in‚ 4.98046875 disk.dm-1.in‚7.3125! disk.rbd2.out‚190.87060546875 disk.sda1.out‚0 disk.sda.out‚ 313.54296875% proc.memory.availableMB‚ 506.21875 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.sda3.out‚ 313.50390625 disk.rbd1.in‚0% disk./.usageGB‚1097.9403610229492 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚26455.39990234375 proc.cpu.threads‚17 network.sent‚ 1.4697889e+07 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 4.98046875# proc.memory.rssMB‚ 2861.77734375 disk.dm-1.out‚ 313.50390625 disk.rbd0.out‚0 disk.rbd3.out‚0$ memory_percent‚96.91028594970703) proc.memory.percent‚17.466902732849126mŽò‚(:„ ÔÒäÔ¤´®x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.235' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.647 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k7fIƒó‚(:ø ÔÒäÔ»õŸ{% disk./.usageGB‚1098.4423599243164 disk.rbd4.out‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚ 4.98046875! disk.rbd2.out‚190.93310546875 disk.rbd0.out‚0( proc.memory.availableMB‚ 505.92578125) proc.memory.percent‚17.46690273284912 cpu‚25.301897540970074 network.recv‚ 2.082957e+06( disk./.usagePercent‚73.7173884290493( disk.nvme0c0n1.out‚26455.75537109375 disk.sda2.in‚0 network.sent‚ 1.4703321e+07 disk.sda.in‚ 7.36328125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 314.37890625 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 314.41796875 disk.sda3.in‚7.3125 disk.sda3.out‚ 314.37890625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 2.97119140625& disk.nvme0n1.out‚26455.75537109375 disk.dm-0.in‚ 4.98046875# memory_percent‚96.9120740890503# proc.memory.rssMB‚ 2861.77734375 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd4.in‚0# disk.dm-0.out‚26455.75537109375 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.9804687583ß2ô- _wandbruntime‚5960  _runtime‚5960ÚçQÄõ¾ % lr-AdamW/pg1‚0.000458927625335459 % lr-AdamW/pg2‚0.000458927625335459 trainer/global_step‚6149 $ _timestamp‚1.788422489480167e+09 _runtime‚5960.230156021 _step‚503mã™êÉöà % lr-AdamW/pg1‚0.000458927625335459 % lr-AdamW/pg2‚0.000458927625335459 trainer/global_step‚6149 $ _timestamp‚1.788422489480167e+09 _runtime‚5960.230156021 _step‚503÷cOª•2÷- _wandbruntime‚5960  _runtime‚5960ý6§øí'øçO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚255456054290944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231070498268160 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66518182 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231070515307520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255456054290944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚242463628560178 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 % _timestamp‚1.7884224894809659e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11393353941166 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41455658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11426530673152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113688843 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261843847200768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160155428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107973840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚273470232249344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273869943 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚273470533399552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11626662258688 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255456300497408 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11426302469632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244029751821312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11426530673152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273844271 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160181098 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚244029769824256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚255456300497408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160155428 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 _step‚504 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113688843 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107973183 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚261843871140864 trainer/global_step‚6149 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113688845 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41455003 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113688845 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255456300497408 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11426302469632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244029751821312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11426530673152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273844271 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160181098 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚244029769824256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚255456300497408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160155428 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 _step‚504 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113688843 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107973183 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚261843871140864 trainer/global_step‚6149 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113688845 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41455003 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113688845 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255456054290944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚242463628560178 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 % _timestamp‚1.7884224894809659e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11393353941166 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41455658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11426530673152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113688843 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261843847200768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160155428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107973840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚273470232249344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273869943 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚273470533399552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11626662258688 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160180979 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11428368380928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107990347 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor¬ïÒFü.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160206675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66528537 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 _runtime‚5961.109216723 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 % _timestamp‚1.7884224903587005e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _step‚505 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255496229929472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107989733 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231106817765376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255496476149248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚273513565495296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231106800726016 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11428368380928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚242501763417634 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255496476149248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11428140164096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273913239 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113706562 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11395186340974 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273913239 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚244068089765376 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261885040837120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113706564 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160206675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113706562 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚244068089765376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11628524658176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113706564 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41461198 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273913239 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113706562 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11395186340974 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273913239 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚244068089765376 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261885040837120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113706564 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160206675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113706562 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚244068089765376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11628524658176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113706564 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41461198 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160180979 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11428368380928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107990347 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160206675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66528537 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 _runtime‚5961.109216723 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 % _timestamp‚1.7884224903587005e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _step‚505 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255496229929472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107989733 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231106817765376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255496476149248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚273513565495296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231106800726016 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11428368380928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚242501763417634 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ùÊqìã2ý- _wandbruntime‚5961  _runtime‚5961˱¶^þØ trainer/global_step‚6149 1loss/FBt1/assignment_loss‚0.21171869395766407 2loss/FBt4/assignment_loss‚0.014214681125142507 0loss/SRqqt2/detection_loss‚1.069575573238781 0loss/FRt2/assignment_loss‚1.8307494177268102 /loss/FBt2/detection_loss‚0.8390786476536758 epoch‚5 2loss/SRqqt2/assignment_loss‚0.9415386506687786 0loss/FBt2/assignment_loss‚0.0673031339085891 1loss/SRqqt1/detection_loss‚2.1563064396381377 /loss/FBt1/detection_loss‚2.3450509707132974 .loss/FBt3/detection_loss‚4.011126667261124 /loss/FBt4/detection_loss‚2.2092446618609958 0loss/FRt1/assignment_loss‚3.5892606956618174 2loss/SRqqt4/assignment_loss‚1.5420214473693208 .loss/FRt1/detection_loss‚2.571440011795078 1loss/SRqqt4/detection_loss‚2.0127019215036523 /loss/FRt4/detection_loss‚2.3303800807757904 /loss/FRt4/assignment_loss‚3.124648317694664 .loss/FRt3/detection_loss‚2.423249895805898 1loss/SRqqt3/detection_loss‚1.8268906792690014 2loss/FBt3/assignment_loss‚0.019855159955720108 % _timestamp‚1.7884224903606205e+09 _runtime‚5961.110437994 _step‚506 /loss/FRt2/detection_loss‚1.8164526199778686 0loss/FRt3/assignment_loss‚2.8987941806731015 1loss/SRqqt1/assignment_loss‚1.722943327948451 2loss/SRqqt3/assignment_loss‚1.9603300052213257 %loss/total_loss‚43.53487588140469à~_¬cÿÝ 2loss/FBt3/assignment_loss‚0.019855159955720108 % _timestamp‚1.7884224903606205e+09 _runtime‚5961.110437994 _step‚506 /loss/FRt2/detection_loss‚1.8164526199778686 0loss/FRt3/assignment_loss‚2.8987941806731015 1loss/SRqqt1/assignment_loss‚1.722943327948451 2loss/SRqqt3/assignment_loss‚1.9603300052213257 %loss/total_loss‚43.53487588140469 trainer/global_step‚6149 1loss/FBt1/assignment_loss‚0.21171869395766407 2loss/FBt4/assignment_loss‚0.014214681125142507 0loss/SRqqt2/detection_loss‚1.069575573238781 0loss/FRt2/assignment_loss‚1.8307494177268102 /loss/FBt2/detection_loss‚0.8390786476536758 epoch‚5 2loss/SRqqt2/assignment_loss‚0.9415386506687786 0loss/FBt2/assignment_loss‚0.0673031339085891 1loss/SRqqt1/detection_loss‚2.1563064396381377 /loss/FBt1/detection_loss‚2.3450509707132974 .loss/FBt3/detection_loss‚4.011126667261124 /loss/FBt4/detection_loss‚2.2092446618609958 0loss/FRt1/assignment_loss‚3.5892606956618174 2loss/SRqqt4/assignment_loss‚1.5420214473693208 .loss/FRt1/detection_loss‚2.571440011795078 1loss/SRqqt4/detection_loss‚2.0127019215036523 /loss/FRt4/detection_loss‚2.3303800807757904 /loss/FRt4/assignment_loss‚3.124648317694664 .loss/FRt3/detection_loss‚2.423249895805898 1loss/SRqqt3/detection_loss‚1.8268906792690014úOßY›€‚(: ãÒäÔÿϤx gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.727' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.945400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]lj2~‚(:ó ãÒäÔÁÃ{ disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 network.recv‚ 2.087353e+06 disk.rbd1.in‚0 disk.sda1.in‚ 0.05078125$ memory_percent‚96.91252708435059 cpu‚25.301223278743244 disk.sda2.out‚ 0.0390625! disk.rbd2.out‚190.94873046875 disk.rbd4.out‚0( disk.nvme0c0n1.out‚26554.95947265625 proc.cpu.threads‚17% disk./.usageGB‚1098.4403648376465& disk.nvme0n1.out‚26554.95947265625# disk.dm-0.out‚26554.95947265625) disk./.usagePercent‚73.71725453710889 disk.nvme0n1.in‚ 4.98046875 disk.dm-0.in‚ 4.98046875 disk.sda2.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 505.8515625 disk.nvme0c0n1.in‚ 4.98046875 disk.sda.out‚314.9375# proc.memory.rssMB‚ 2861.77734375 disk.sda3.in‚7.3125 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0) proc.memory.percent‚17.46690273284912 disk.sda3.out‚ 314.8984375 disk.rbd1.out‚ 2.97119140625 disk.drbd1777.in‚ 1.03515625 network.sent‚ 1.4709354e+07 disk.dm-1.out‚ 314.8984375 disk.rbd4.in‚0 disk.sda1.out‚0}å‚‚(:… òÒäÔø°x gpu.0.gpu‚21.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ʘ‰lƒ‚(:á òÒäÔâ‡Íz disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 315.6328125 disk.sda3.in‚7.3125# disk.nvme0n1.out‚26654.18359375# memory_percent‚96.9125747680664 proc.cpu.threads‚17 disk.rbd0.out‚0 network.recv‚ 2.093346e+06 disk.dm-0.out‚26654.18359375 disk.dm-1.out‚ 315.59375 disk.nvme0c0n1.in‚ 4.98046875 disk.sda.in‚ 7.36328125 network.sent‚ 1.4817904e+07 disk.drbd1777.in‚ 1.03515625# proc.memory.rssMB‚ 2861.77734375 cpu‚25.30060447118624 disk.rbd0.in‚0 disk.dm-0.in‚ 4.98046875 disk.drbd1777.out‚0% disk.nvme0c0n1.out‚26654.18359375 disk.sda1.out‚0 disk.rbd1.in‚0) proc.memory.percent‚17.46690273284912% disk./.usageGB‚1098.4403495788574 disk.sda3.out‚ 315.59375) disk./.usagePercent‚73.71725351307875 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd2.out‚ 191.0390625 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 4.98046875 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 3.00341796875% proc.memory.availableMB‚ 505.84375£‘ŒÀ„‚(:… ÓäÔ‡îªx gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚100.063' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.0126 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õæˆ~…‚(:ó ÓäÔûÞÓz disk.rbd2.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 network.sent‚ 1.4824321e+07# disk./.usageGB‚1098.44038772583# disk.dm-0.out‚26753.69384765625 disk.drbd1777.out‚0 disk.sda.in‚ 7.36328125# proc.memory.rssMB‚ 2861.77734375 disk.nvme0n1.in‚ 4.98046875 disk.rbd4.in‚0 disk.rbd1.out‚ 3.00341796875 cpu‚25.299938770106106 disk.sda2.out‚ 0.0390625 disk.sda1.in‚ 0.05078125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚26753.69384765625) disk./.usagePercent‚73.71725607315409 disk.sda2.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.98046875 disk.sda3.out‚ 316.08203125 disk.rbd1.in‚0& disk.nvme0n1.out‚26753.69384765625 disk.dm-0.in‚ 4.98046875 disk.rbd2.out‚ 191.0390625 disk.rbd3.in‚0) proc.memory.percent‚17.46690273284912 network.recv‚ 2.098149e+06 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.dm-1.in‚7.3125 disk.sda.out‚ 316.12109375$ memory_percent‚96.91414833068848' proc.memory.availableMB‚ 505.5859375 disk.sda1.out‚0 disk.dm-1.out‚ 316.08203125©Š÷ß2†- _wandbruntime‚6006  _runtime‚6006½þ¦wLJÁ & lr-AdamW/pg1‚0.0004531909733283291 & lr-AdamW/pg2‚0.0004531909733283291 trainer/global_step‚6199 % _timestamp‚1.7884225353247516e+09 _runtime‚6006.074676949 _step‚507ðË»ÌˆÆ _step‚507 & lr-AdamW/pg1‚0.0004531909733283291 & lr-AdamW/pg2‚0.0004531909733283291 trainer/global_step‚6199 % _timestamp‚1.7884225353247516e+09 _runtime‚6006.074676949ûL&zd2‰- _wandbruntime‚6006  _runtime‚6006(xDÁ‹ŠæO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67035930 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114574795 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114574793 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚257465096837632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚ 6006.07562747 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161436330 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚6199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161462000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276036795 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11484763863226 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚257464850631168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11518429810176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108812037 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41776765 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114574795 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114574793 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108812697 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚275622566511616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263902332013568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161462000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67035932 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 % _timestamp‚1.7884225353254735e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11518429810176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263902355953664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚245946667027456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11719935444992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 _step‚508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41776107 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚245946649024512 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11518201606656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚275622267458560 YEDeviceStatsMonimЀ\a tor.on_train_batch_start/allocated_bytes.all.allocated‚257465096837632 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11518201606656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚232885638200320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11720210557952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11484987512374 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚245946667027456 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232885621160960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚244370625712694 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚257464850631168#áÈñ'‹ëO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161436330 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚6199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161462000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276036795 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11484763863226 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚257464850631168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11518429810176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108812037 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41776765 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114574795 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114574793 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108812697 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚275622566511616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263902332013568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161462000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67035932 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 % _timestamp‚1.7884225353254735e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11518429810176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263902355953664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚245946667027456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11719935444992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 _step‚508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41776107 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚245946649024512 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11518201606656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚275622267458560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚257465096837632 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11518201606656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚232885638200320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11720210557952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11484987512374 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚245946667027456 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232885621160960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚244370625712694 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚257464850631168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67035930 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114574795 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114574793 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚257465096837632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚ 6006.07562747 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670üпѻ2Œ- _wandbruntime‚6006  _runtime‚6006SIž÷&ñM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276054393 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚257505026269696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 305331200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232921940658176 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚509 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276080091 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚263943501709824 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 % _timestamp‚1.7884225361812382e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚245985004971520 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161461881 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚244408519881642 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚257505272489472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276054393 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67046287 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11721811046912 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108828961 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41782676 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11486596262826 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161461881 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚245984986968576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263943525649920 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11486819912182 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276080091 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232921940658176 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚509 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276080091 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚263943501709824 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 % _timestamp‚1.7884225361812382e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚245985004971520 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161461881 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚244408519881642 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚257505272489472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276054393 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67046287 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11721811046912 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108828961 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41782676 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11486596262826 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161461881 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚245984986968576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263943525649920 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11486819912182 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276080091 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11610328947200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 % _timestamp‚1.7884225813619058e+09 _step‚512 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11610328947200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11610100743680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67553682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42104424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚247863564230656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11813801750528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162717232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109657434 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162742902 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259473893177856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259473646971392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42103754 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162742902 trainer/global_step‚6249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚247863546227712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246277382176702 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115460743 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162717232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚2659608168263687t¦aò'ìO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278203647 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115460745 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚247863546227712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11610328947200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 % _timestamp‚1.7884225813619058e+09 _step‚512 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11610328947200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11610100743680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67553682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42104424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚247863564230656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11813801750528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162717232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109657434 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162742902 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259473893177856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259473646971392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42103754 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162742902 trainer/global_step‚6249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚247863546227712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246277382176702 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115460743 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162717232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚265960816826368 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11576621083582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚247863564230656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234700744053760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259473646971392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234700761093120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67553680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚277774349755392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚265960840766464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚246277141488194 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11610100743680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109658106 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504€êg{_2ž- _wandbruntime‚6052  _runtime‚6052¹Ö_RÆŸðM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162742783 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚247901884171776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278246943 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚247901902174720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11578453483390 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚247901902174720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42110451 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚277817383846912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 $ _timestamp‚1.788422582246902e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11612166654976 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259514068829696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266001986522624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 _step‚513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚247901884171776 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109674486 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115478462 VCDeviceStatsMonitor.on_trai5Í0n_batch_end/active_bytes.small_pool.freed‚11611938438144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115478464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266002010462720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67564037 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162768479 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11612166654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259513822609920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚246315517034366 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259514068829696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246315276345650 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109675118 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚234737063550976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278221245 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162768479 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚247901902174720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11578453483390 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚247901902174720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42110451 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚277817383846912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 $ _timestamp‚1.788422582246902e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11612166654976 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259514068829696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266001986522624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 _step‚513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚247901884171776 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109674486 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115478462 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11611938438144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115478464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266002010462720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67564037 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162768479 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11612166654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259513822609920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚246315517034366 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259514068829696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246315276345650 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109675118 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚234737063550976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278221245 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162768479 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278246943 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261482443311616 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚261482689518080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116346693 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163998134 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249780443430912 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280344827 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11702228084224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11701999880704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚679 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11907004740096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110499392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42428641 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚248184138640710 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _step‚516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42427962 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚236515866946560 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261482689518080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279926306379264 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110500073 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68071430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164023804 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68071432 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11907279853056 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚248184138640710 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _step‚516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42427962 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚236515866946560 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261482689518080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279926306379264 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110500073 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68071430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164023804 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68071432 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11907279853056 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261482443311616 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚261482689518080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116346693 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163998134 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249780443430912 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280344827 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11702228084224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11701999880704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚679 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11907004740096„–Z¾¿2°- _wandbruntime‚6099  _runtime‚6099;2~A÷&±ñM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11704065792000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚236552186443776 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 _runtime‚6099.239794525 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68081787 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268060495275520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚261522865169920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11908874050560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚248222032809658 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249818799377920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164049381 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚261522618950144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280388097 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280413795 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164049381 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110516630 trainer/global_step‚6299 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884226284891608e+09 _step‚517 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249818781374976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11703837575168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164023685 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚654 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116364414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚268060471335424 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110515976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249818799377920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164023685 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279969638134272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚248222273498374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249818781374976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116364414 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11703837575168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280413795 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚279969345385984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11704065792000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11670087054598 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116364412 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261522865169920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280388097 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116364412 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11909142858752 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚236552169404416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ðóü&²öM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68081785 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42434191 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11703837575168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280413795 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚279969345385984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11704065792000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11670087054598 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116364412 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261522865169920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280388097 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116364412 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11909142858752 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚236552169404416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11704065792000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚236552186443776 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 _runtime‚6099.239794525 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68081787 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268060495275520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚261522865169920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11908874050560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚248222032809658 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249818799377920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164049381 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚261522618950144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280388097 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280413795 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164049381 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110516630 trainer/global_step‚6299 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884226284891608e+09 _step‚517 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249818781374976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11703837575168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164023685 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚654 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116364414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚268060471335424 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110515976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249818799377920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164023685 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279969638134272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚248222273498374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249818781374976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116364414 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚251697358637056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42753437 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11794127221248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263491239651840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42754092 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚238331006878720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚238330989839360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 282537351 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111342617 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282511679 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚263491485858304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68589180 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165304706 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚250090654416210 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117232645 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117232643 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11793899017728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0gu‡ò'ÁìO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚251697358637056 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11794127221248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚251697358637056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42753437 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11794127221248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263491239651840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42754092 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚238331006878720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚238330989839360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 282537351 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111342617 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282511679 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚263491485858304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68589180 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165304706 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚250090654416210 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117232645 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117232643 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11793899017728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ˆ)”·Ö2Â- _wandbruntime‚6145  _runtime‚6145è÷±1tÃñM OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117250362 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚660 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165304587 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117250364 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165304587 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚282121493805056 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚251735696581120 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111359088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚270118980088320 _runtime‚6145.101987716 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117250364 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263531661510144 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282580647 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282554949 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H@DeviceStatsMonitor.on_train_batch_end/'Zsƒsegment.large_pool.current‚299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚662 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117250362 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚251735678578176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165330283 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚250129029962382 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚251735678578176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11795964929024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12002242811392 trainer/global_step‚6349 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165330283 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42759553 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832CvŒü&ÄöM PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165330283 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42759553 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117250362 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚660 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165304587 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117250364 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165304587 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚282121493805056 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚251735696581120 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111359088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚270118980088320 _runtime‚6145.101987716 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117250364 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263531661510144 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282580647 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282554949 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚662 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117250362 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚251735678578176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165330283 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚250129029962382 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚251735678578176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11795964929024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12002242811392 trainer/global_step‚6349‰ÒUÝÌ2Å- _wandbruntime‚6145  _runtime‚6145‚û[ÆÕ 2loss/SRqqt2/assignment_loss‚1.0363072911635614 /loss/FRt4/detection_loss‚2.5874052950314113 1loss/SRqqt3/detection_loss‚1.6761837822850794 /loss/FBt2/detection_loss‚0.6404445224695585 .loss/FBt4/detection_loss‚3.280298113822937 $loss/total_loss‚43.0867784491253 trainer/global_step‚6349 /loss/FRt1/assignment_loss‚3.347271066336405 1loss/SRqqt1/detection_loss‚2.2761509858426594 0loss/FRt2/assignment_loss‚1.4192837381362915 2loss/SRqqt1/assignment_loss‚2.0524700707977726 .loss/FRt2/detection_loss‚1.848001570403576 1loss/SRqqt2/detection_loss‚1.3468390055932105 1loss/SRqqt4/detection_loss‚1.7864725166665656 _runtime‚6145.103032795 2loss/SRqqt3/assignment_loss‚1.7476688243914396 1loss/SRqqt4/assignment_loss‚1.505028901987576 0loss/FBt4/assignment_loss‚0.1378360234084539 /loss/FBt1/detection_loss‚1.9304597228765488 /loss/FRt3/detection_loss‚2.2754996006793164 _step‚522 1loss/FBt1/assignment_loss‚0.09281035830887656 1loss/FBt2/assignment_loss‚0.10125675376221845 1loss/FBt3/assignment_loss‚0.13503686932381243 /loss/FBt3/detection_loss‚2.5947723910212517 /loss/FRt4/assignment_loss‚3.453763823849814 epoch‚5 /loss/FRt3/assignment_loss‚3.134939064370825 /loss/FRt1/detection_loss‚2.6805781565961384 % _timestamp‚1.7884226743532102e+09NåüÊ`ÇÚ 0loss/FRt2/assignment_loss‚1.4192837381362915 2loss/SRqqt1/assignment_loss‚2.0524700707977726 .loss/FRt2/detection_loss‚1.848001570403576 1loss/SRqqt2/detection_loss‚1.3468390055932105 1loss/SRqqt4/detection_loss‚1.7864725166665656 _runtime‚6145.103032795 2loss/SRqqt3/assignment_loss‚1.7476688243914396 1loss/SRqqt4/assignment_loss‚1.505028901987576 0loss/FBt4/assignment_loss‚0.1378360234084539 /loss/FBt1/detection_loss‚1.9304597228765488 /loss/FRt3/detection_loss‚2.2754996006793164 _step‚522 1loss/FBt1/assignment_loss‚0.09281035830887656 1loss/FBt2/assignment_loss‚0.10125675376221845 1loss/FBt3/assignment_loss‚0.13503686932381243 /loss/FBt3/detection_loss‚2.5947723910212517 /loss/FRt4/assignment_loss‚3.453763823849814 epoch‚5 /loss/FRt3/assignment_loss‚3.134939064370825 /loss/FRt1/detection_loss‚2.6805781565961384 % _timestamp‚1.7884226743532102e+09 2loss/SRqqt2/assignment_loss‚1.0363072911635614 /loss/FRt4/detection_loss‚2.5874052950314113 1loss/SRqqt3/detection_loss‚1.6761837822850794 /loss/FBt2/detection_loss‚0.6404445224695585 .loss/FBt4/detection_loss‚3.280298113822937 $loss/total_loss‚43.0867784491253 trainer/global_step‚6349 /loss/FRt1/assignment_loss‚3.347271066336405 1loss/SRqqt1/detection_loss‚2.2761509858426594ŠXÿ ¶È‚(:„ —ÔäÔÀœ¯x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.717' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9434 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0} M£É‚(:ö —ÔäÔ«¾Î{ disk.sda3.in‚7.3125% disk.nvme0n1.out‚27449.7529296875 network.sent‚ 1.5190765e+07' disk.nvme0c0n1.out‚27449.7529296875 disk.sda1.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 4.98046875 disk.sda2.out‚ 0.0390625' proc.memory.availableMB‚ 338.0546875 proc.cpu.threads‚17 disk.sda1.in‚ 0.05078125 disk.rbd1.in‚0$ memory_percent‚97.93667793273926) disk./.usagePercent‚73.71735975620545! disk.rbd2.out‚194.57568359375 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 321.77734375 disk.sda3.out‚ 321.73828125) proc.memory.percent‚17.46690273284912 cpu‚25.293820214637297 disk.rbd1.out‚ 3.12939453125 network.recv‚ 2.145118e+06 disk.dm-1.out‚ 321.73828125 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda2.in‚0# proc.memory.rssMB‚ 2861.77734375% disk./.usageGB‚1098.4419326782227 disk.rbd4.out‚0 disk.nvme0c0n1.in‚ 4.98046875 disk.sda.in‚ 7.36328125" disk.dm-0.out‚27449.7529296875 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.dm-0.in‚ 4.98046875@ù^T›Ê‚(: ¦ÔäÔ¼»¨x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úpó:|Ë‚(:ñ ¦ÔäÔý‘…{ network.recv‚ 2.150558e+06 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 322.234375% disk.nvme0n1.out‚27548.9560546875 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 4.98046875 cpu‚25.293234355586634 disk.dm-1.out‚ 322.234375 disk.rbd0.in‚0 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.12939453125 disk.nvme0n1.in‚ 4.98046875 disk.dm-0.in‚ 4.98046875 disk.rbd3.in‚0 disk.sda2.in‚0$ memory_percent‚97.93658256530762' proc.memory.availableMB‚ 338.0703125 proc.cpu.threads‚17! disk.rbd2.out‚194.71630859375 disk.rbd4.out‚0 disk.drbd1777.out‚0 network.sent‚ 1.5299127e+07 disk.rbd3.out‚0 disk.sda.out‚ 322.2734375' disk.nvme0c0n1.out‚27548.9560546875 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125% disk./.usageGB‚1098.4410820007324 disk.sda3.in‚7.3125" disk.dm-0.out‚27548.9560546875 disk.rbd4.in‚0) disk./.usagePercent‚73.71730266652531 disk.sda1.out‚0# proc.memory.rssMB‚ 2861.77734375) proc.memory.percent‚17.46690273284912 disk.drbd1777.in‚ 1.03515625 disk.rbd0.out‚0xµrXšÌ‚(: µÔäÔŸ„x gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.751' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.750200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b<ÜärÍ‚(:ç µÔäÔÃì{ disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625) proc.memory.percent‚17.46692657470703 disk.rbd2.in‚0 network.sent‚ 1.5304725e+07 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚27648.36572265625 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 disk.drbd1777.out‚0# disk./.usageGB‚1098.44144821167 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 322.80859375 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚27648.36572265625 disk.sda2.in‚0 disk.dm-1.out‚ 322.80859375( proc.memory.availableMB‚ 337.77734375 proc.memory.rssMB‚ 2861.78125 cpu‚25.29263985789802 disk.sda1.out‚0 disk.rbd1.out‚ 3.12939453125! disk.rbd2.out‚196.22802734375 disk.dm-1.in‚7.3125 network.recv‚ 2.154864e+06 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125& disk.nvme0n1.out‚27648.36572265625$ memory_percent‚97.93837070465088 disk.sda.out‚ 322.84765625 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17( disk./.usagePercent‚73.7173272432486 disk.rbd4.in‚0 disk.sda3.in‚7.3125 disk.rbd3.out‚06›DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12094232462336 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265500035992064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚284230230711296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11885798154752 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166559938 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272136295204864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚284230527667200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251997410880218 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11851298148058 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166559938 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43082064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚265500282198528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284704203 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284678531 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272136271264768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240146112732160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚253614237837312 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 112188346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69106930 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12093959446528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240146129771520 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112188996 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11851521797206 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11886026358272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12094232462336 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265500035992064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚284230230711296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11885798154752 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166559938 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272136295204864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚284230527667200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251997410880218 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11851298148058 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166559938 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43082064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚265500282198528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284704203 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284678531 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272136271264768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240146112732160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚253614237837312 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 112188346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69106930 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12093959446528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240146129771520 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112188996 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11851521797206 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11886026358272 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112205532 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚265540211630592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 trainer/global_step‚6399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚253652575781376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284721801 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11853130547658 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272177464901120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚253652593784320 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚252035545737674 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚284273257438720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112206178 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚265540457850368 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚253652575781376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11887635849216 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118136312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240182415190016 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272177440961024 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284721801 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚525 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284747499 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚265540457850368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118136314 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118136312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚253652593784320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284747499 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43088247 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43088891 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 % _timestamp‚1.7884227204858158e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166611185 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 283488256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6191.236572839 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11887635849216\7àøü&ÖöM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166611185 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚252035545737674 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚284273257438720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112206178 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚265540457850368 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚253652575781376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11887635849216 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118136312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240182415190016 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272177440961024 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284721801 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚525 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284747499 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚265540457850368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118136314 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118136312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚253652593784320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284747499 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43088247 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43088891 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 % _timestamp‚1.7884227204858158e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166611185 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 283488256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6191.236572839 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11887635849216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12096099965952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚252035786426390 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112205532 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚265540211630592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 trainer/global_step‚6399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚253652575781376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284721801 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11853130547658 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272177464901120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚253652593784320 F5óÞ‚(:„ âÔäÔ¬­Éx gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žPlêXß‚(:Í âÔäÔ¤Å{ disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 3.12939453125 disk.dm-0.in‚5.3125 disk.dm-1.in‚7.3125 network.recv‚ 2.168758e+06 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.rbd1.in‚0 disk.dm-1.out‚ 325.03125 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.dm-0.out‚ 27847.6484375 disk.sda2.in‚0$ proc.memory.availableMB‚339.0625 cpu‚25.2908956334179 network.sent‚ 1.542449e+07 disk.sda1.out‚0 disk.sda3.out‚ 325.03125 disk.nvme0n1.in‚5.3125 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 27847.6484375" disk.nvme0n1.out‚ 27847.6484375 disk.sda2.out‚ 0.0390625 proc.memory.rssMB‚ 2861.78125# disk./.usageGB‚1098.44136428833 disk.rbd2.out‚ 196.541015625) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17) disk./.usagePercent‚73.71732161108285 disk.rbd2.in‚0 disk.sda.out‚ 325.0703125$ memory_percent‚97.93052673339844m@Ò2à- _wandbruntime‚6236  _runtime‚6236aÔSÈá ' lr-AdamW/pg1‚0.00042436904760213313 ' lr-AdamW/pg2‚0.00042436904760213313 trainer/global_step‚6449 % _timestamp‚1.7884227654742212e+09 _runtime‚ 6236.22418365 _step‚527|lô±ÍâÇ trainer/global_step‚6449 % _timestamp‚1.7884227654742212e+09 _runtime‚ 6236.22418365 _step‚527 ' lr-AdamW/pg1‚0.00042436904760213313 ' lr-AdamW/pg2‚0.00042436904760213313wÙ˜2ã- _wandbruntime‚6236  _runtime‚6236çí'äçO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167866510 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255531135040512 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚6449 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286845383 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167840840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚241961252664320 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267508832332288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69624680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286382141420544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43408361 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167866510 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚253904408032734 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11977925495296 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267509078538752 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11977925495296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267509078538752 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113033668 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12187654164480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11942931719266 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _step‚528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69624682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119004543 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚241961235624960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286871055 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267508832332288 _runtime‚6236.225345589pÝ›ò'åìO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11977925495296 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267509078538752 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11977925495296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267509078538752 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113033668 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12187654164480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11942931719266 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _step‚528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69624682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119004543 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚241961235624960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286871055 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267508832332288 _runtime‚6236.225345589 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167866510 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255531135040512 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚6449 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286845383 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167840840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚241961252664320 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267508832332288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69624680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286382141420544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43408361 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167866510 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚253904408032734 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299;ˆU2æ- _wandbruntime‚6237  _runtime‚6237AyùF†çñM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286914351 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113050261 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167892087 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11979763203072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69635037 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274235949713920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119022264 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11944764118866 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119022262 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113050897 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43415860 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267549007970816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11944987768222 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚529 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12189525444096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253942542890398 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228Á7Øq 216832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286888653 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚241997555122176 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267549254190592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286914351 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43415226 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286888653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69635035&øIü&èöM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11944764118866 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119022262 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113050897 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43415860 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267549007970816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11944987768222 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚529 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12189525444096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253942542890398 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286888653 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚241997555122176 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267549254190592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286914351 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43415226 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286888653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69635035 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286914351 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113050261 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167892087 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11979763203072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69635037 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274235949713920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119022264 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432‘Þ$Ÿî2é- _wandbruntime‚6237  _runtime‚6237.‚!]VêÐ 0loss/FRt1/assignment_loss‚4.2314521289534035 /loss/FRt2/assignment_loss‚1.534084689617157 .loss/FRt1/detection_loss‚2.811818433718549 /loss/FBt2/detection_loss‚0.5800944515631521 .loss/FBt4/detection_loss‚2.126660504124381 2loss/SRqqt3/assignment_loss‚1.4886488834023475 0loss/SRqqt3/detection_loss‚1.811778548359871 1loss/SRqqt1/assignment_loss‚2.334403020995004 .loss/FRt3/detection_loss‚2.473840943309996 1loss/SRqqt1/detection_loss‚2.6742936947516034 _step‚530 1loss/FBt2/assignment_loss‚0.15978395612467333 0loss/FBt3/assignment_loss‚0.1471329106716439 /loss/FBt3/detection_loss‚2.9226195588707924 %loss/total_loss‚44.72685649973813 epoch‚5 trainer/global_step‚6449 _runtime‚6237.109748904 /loss/FRt3/assignment_loss‚2.748527530829112 0loss/FBt4/assignment_loss‚0.9827384917615828 % _timestamp‚1.7884227663599358e+09 0loss/FRt4/assignment_loss‚3.5492686162123808 2loss/SRqqt4/assignment_loss‚1.6113054948332517 1loss/FBt1/assignment_loss‚0.25468334849786817 .loss/FRt2/detection_loss‚1.471658219397068 .loss/FRt4/detection_loss‚2.750418588116362 0loss/SRqqt4/detection_loss‚2.059625572808411 /loss/FBt1/detection_loss‚2.3627519657214484 2loss/SRqqt2/assignment_loss‚0.8496124528896283 0loss/SRqqt2/detection_loss‚0.789654494208448Uyc6[ëÕ 1loss/SRqqt1/assignment_loss‚2.334403020995004 .loss/FRt3/detection_loss‚2.473840943309996 1loss/SRqqt1/detection_loss‚2.6742936947516034 _step‚530 1loss/FBt2/assignment_loss‚0.15978395612467333 0loss/FBt3/assignment_loss‚0.1471329106716439 /loss/FBt3/detection_loss‚2.9226195588707924 %loss/total_loss‚44.72685649973813 epoch‚5 trainer/global_step‚6449 _runtime‚6237.109748904 /loss/FRt3/assignment_loss‚2.748527530829112 0loss/FBt4/assignment_loss‚0.9827384917615828 % _timestamp‚1.7884227663599358e+09 0loss/FRt4/assignment_loss‚3.5492686162123808 2loss/SRqqt4/assignment_loss‚1.6113054948332517 1loss/FBt1/assignment_loss‚0.25468334849786817 .loss/FRt2/detection_loss‚1.471658219397068 .loss/FRt4/detection_loss‚2.750418588116362 0loss/SRqqt4/detection_loss‚2.059625572808411 /loss/FBt1/detection_loss‚2.3627519657214484 2loss/SRqqt2/assignment_loss‚0.8496124528896283 0loss/SRqqt2/detection_loss‚0.789654494208448 0loss/FRt1/assignment_loss‚4.2314521289534035 /loss/FRt2/assignment_loss‚1.534084689617157 .loss/FRt1/detection_loss‚2.811818433718549 /loss/FBt2/detection_loss‚0.5800944515631521 .loss/FBt4/detection_loss‚2.126660504124381 2loss/SRqqt3/assignment_loss‚1.4886488834023475 0loss/SRqqt3/detection_loss‚1.811778548359871’Þ‘ÚÌì‚(:… ñÔäÔ®Á›x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.011' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÀL«eí‚(:Ú ñÔäÔ¥žÛz disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 cpu‚25.29032600599701# disk./.usageGB‚1098.44136428833 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚325.5625 disk.rbd1.in‚0! disk.dm-0.out‚27947.111328125 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 325.5234375$ memory_percent‚97.93040752410889) disk./.usagePercent‚73.71732161108285 disk.dm-1.out‚ 325.5234375 disk.rbd4.in‚0 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125& disk.nvme0c0n1.out‚27947.111328125 disk.sda3.in‚7.3125 disk.nvme0n1.in‚5.3125 disk.rbd2.out‚ 196.541015625 disk.rbd0.in‚0 disk.rbd1.out‚ 3.12939453125$ disk.nvme0n1.out‚27947.111328125 network.sent‚ 1.5430843e+07 disk.dm-0.in‚5.3125 disk.sda.in‚ 7.36328125 proc.cpu.threads‚17 network.recv‚ 2.173573e+06 disk.sda1.in‚ 0.05078125( proc.memory.availableMB‚ 339.08203125¿Ø3 ›î‚(: €ÕäÔ›¸x gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.252' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.650399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰NÙUcï‚(:Ø €ÕäÔÉÇûz disk.dm-0.in‚5.3125 disk.rbd2.out‚ 198.255859375) disk./.usagePercent‚73.71732161108285 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 cpu‚25.289710298819518 disk.rbd3.out‚0 disk.dm-0.out‚28046.28515625' proc.memory.availableMB‚ 339.2578125 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 disk.rbd2.in‚0 network.recv‚ 2.179089e+06 disk.sda1.out‚0# disk.nvme0n1.out‚28046.28515625 disk.sda2.in‚0 disk.dm-1.out‚ 325.953125 proc.cpu.threads‚17 disk.sda3.in‚7.3125 disk.rbd1.out‚ 3.12939453125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0$ memory_percent‚97.92933464050293 disk.sda.in‚ 7.36328125 disk.sda.out‚ 325.9921875# disk./.usageGB‚1098.44136428833 disk.sda1.in‚ 0.05078125 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 325.953125 network.sent‚ 1.5538701e+07 disk.nvme0c0n1.in‚5.3125% disk.nvme0c0n1.out‚28046.28515625 disk.sda2.out‚ 0.0390625 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125U (Ðð‚(:„ ÕäÔò±x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.901' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äu°¢mñ‚(:â ÕäÔó¼{ disk.rbd3.in‚0 disk.sda.out‚ 326.5234375 disk.dm-1.out‚ 326.484375 disk.rbd0.in‚0 network.recv‚ 2.183867e+06) disk./.usagePercent‚73.71494227705962 disk.rbd1.in‚0 disk.rbd2.in‚0 proc.memory.rssMB‚ 2861.78125 proc.cpu.threads‚17 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 326.484375 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0( disk.nvme0c0n1.out‚28146.06689453125 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125& disk.nvme0n1.out‚28146.06689453125 disk.dm-1.in‚7.3125 disk.rbd0.out‚0% disk./.usageGB‚1098.4059104919434 disk.rbd4.out‚0 disk.rbd2.out‚ 198.255859375$ memory_percent‚97.93095588684082 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0 disk.sda2.out‚ 0.0390625' proc.memory.availableMB‚ 338.9921875) proc.memory.percent‚17.46692657470703 disk.rbd1.out‚ 3.12939453125 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚28146.06689453125 cpu‚25.28917455788546 disk.sda1.out‚0 disk.dm-0.in‚5.3125 network.sent‚ 1.5544638e+07 disk.drbd1777.out‚0”¦{2ò- _wandbruntime‚6282  _runtime‚6282_¡ÇóÁ & lr-AdamW/pg2‚0.0004185853386938932 trainer/global_step‚6499 % _timestamp‚1.7884228117045474e+09 _runtime‚6282.454486643 _step‚531 & lr-AdamW/pg1‚0.0004185853386938932˜áÌôÆ % _timestamp‚1.7884228117045474e+09 _runtime‚6282.454486643 _step‚531 & lr-AdamW/pg1‚0.0004185853386938932 & lr-AdamW/pg2‚0.0004185853386938932 trainer/global_step‚6499“üUÌa2õ- _wandbruntime‚6282  _runtime‚6282QÉ÷í'öçO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚257448032243712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚257448050246656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169147412 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257448050246656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12281164912128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884228117052908e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289012235 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12034565290474 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚276253240890368 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚255811164496742 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169121742 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269517874878976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚257448032243712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119890493 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289037907 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288534429742592 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255810923808234 _runtime‚6282.455401016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12069824632320 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 trainer/global_step‚6499 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169147412 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43734449 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276253264830464 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12280885604864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243776375557120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269517628672512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12069596428800 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119890495 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113876199 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 _step‚532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119890493 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624‰U÷f÷ìO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚257448050246656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169147412 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257448050246656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12281164912128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884228117052908e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289012235 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12034565290474 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚276253240890368 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚255811164496742 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169121742 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269517874878976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚257448032243712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119890493 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289037907 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288534429742592 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255810923808234 _runtime‚6282.455401016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12069824632320 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 trainer/global_step‚6499 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169147412 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43734449 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276253264830464 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12280885604864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243776375557120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269517628672512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12069596428800 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119890495 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113876199 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 _step‚532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119890493 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚257448032243712”®ªà 2ø- _wandbruntime‚6283  _runtime‚6283"7Ëõ&ùïM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119908214 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚257486388190720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12283024722944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113893379 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288577459249664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289081203 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12036621339430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257486388190720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119908212 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257486370187776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113892758 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269557804311040 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289081203 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚276294434526720 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255849058665690 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269558050530816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 6283.31659834 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12071434123264 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚255849299354406 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚243812660975616 $ _timestamp‚1.788422812565856e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119908214 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70152787 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289055505 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119908212 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169172989 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚243812678014976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276294410586624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12282751720448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43739973 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚269558050530816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚257486370187776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70152785 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12036397690074 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360iòy¢ú&úôM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚257486388190720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12283024722944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113893379 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288577459249664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289081203 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12036621339430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257486388190720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119908212 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257486370187776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113892758 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269557804311040 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289081203 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚276294434526720 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255849058665690 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269558050530816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 6283.31659834 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12071434123264 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚255849299354406 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚243812660975616 $ _timestamp‚1.788422812565856e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119908214 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70152787 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289055505 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119908212 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169172989 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚243812678014976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276294410586624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12282751720448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43739973 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚269558050530816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚257486370187776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70152785 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12036397690074 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119908214 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2•ÚÝyG2û- _wandbruntime‚6283  _runtime‚6283°Üß\üÖ 0loss/FRt2/assignment_loss‚1.4092541685644187 /loss/FRt1/detection_loss‚2.5683415651321413 .loss/FBt4/detection_loss‚2.132016936937968 /loss/FRt1/assignment_loss‚3.467531374522618 2loss/SRqqt4/assignment_loss‚1.5994027929428296 1loss/FBt1/assignment_loss‚0.31866185884508824 1loss/SRqqt2/detection_loss‚0.9740179881844069 ,loss/FBt1/detection_loss‚2.5944837530454 /loss/FRt4/detection_loss‚2.4110662566155803 1loss/SRqqt3/detection_loss‚1.6646871344609695 epoch‚5 2loss/SRqqt3/assignment_loss‚1.3081998988647352 1loss/FBt2/assignment_loss‚0.13884895637950245 /loss/FBt3/detection_loss‚2.2350157731109195 &loss/total_loss‚41.752644498627745 _step‚534 /loss/FRt4/assignment_loss‚3.410207933309127 1loss/SRqqt1/assignment_loss‚1.812567333514626 0loss/FBt3/assignment_loss‚0.6120131893290414 1loss/SRqqt1/detection_loss‚2.1815938893202187 1loss/SRqqt4/detection_loss‚2.2080540699072375 0loss/FRt3/assignment_loss‚2.9867794721022896 1loss/FBt4/assignment_loss‚0.20045640423066086 /loss/FBt2/detection_loss‚0.6005486210430914 /loss/FRt2/detection_loss‚1.5557778103733964 /loss/FRt3/detection_loss‚2.4951346432385235 trainer/global_step‚6499 _runtime‚6283.317646044 2loss/SRqqt2/assignment_loss‚0.8679826746529509 % _timestamp‚1.7884228125677948e+09ÆÙõaýÛ ,loss/FBt1/detection_loss‚2.5944837530454 /loss/FRt4/detection_loss‚2.4110662566155803 1loss/SRqqt3/detection_loss‚1.6646871344609695 epoch‚5 2loss/SRqqt3/assignment_loss‚1.3081998988647352 1loss/FBt2/assignment_loss‚0.13884895637950245 /loss/FBt3/detection_loss‚2.2350157731109195 &loss/total_loss‚41.752644498627745 _step‚534 /loss/FRt4/assignment_loss‚3.410207933309127 1loss/SRqqt1/assignment_loss‚1.812567333514626 0loss/FBt3/assignment_loss‚0.6120131893290414 1loss/SRqqt1/detection_loss‚2.1815938893202187 1loss/SRqqt4/detection_loss‚2.2080540699072375 0loss/FRt3/assignment_loss‚2.9867794721022896 1loss/FBt4/assignment_loss‚0.20045640423066086 /loss/FBt2/detection_loss‚0.6005486210430914 /loss/FRt2/detection_loss‚1.5557778103733964 /loss/FRt3/detection_loss‚2.4951346432385235 trainer/global_step‚6499 _runtime‚6283.317646044 2loss/SRqqt2/assignment_loss‚0.8679826746529509 % _timestamp‚1.7884228125677948e+09 0loss/FRt2/assignment_loss‚1.4092541685644187 /loss/FRt1/detection_loss‚2.5683415651321413 .loss/FBt4/detection_loss‚2.132016936937968 /loss/FRt1/assignment_loss‚3.467531374522618 2loss/SRqqt4/assignment_loss‚1.5994027929428296 1loss/FBt1/assignment_loss‚0.31866185884508824 1loss/SRqqt2/detection_loss‚0.9740179881844069–A‹¸ ›þ‚(: žÕäÔªŽx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NÞŠÿ‚(:è žÕäÔèŒõz cpu‚25.288610789913598% disk./.usageGB‚1098.4059104919434 disk.sda.in‚ 7.36328125 disk.rbd4.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 339.0078125 disk.rbd3.out‚0 disk.sda.out‚ 327.03515625 disk.sda3.out‚ 326.99609375 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd2.out‚ 198.333984375 network.sent‚ 1.5~øTé550561e+07 network.recv‚ 2.188186e+06) disk./.usagePercent‚73.71494227705962# disk.dm-0.out‚28146.41064453125 disk.sda2.out‚ 0.0390625 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚28146.41064453125 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚28146.41064453125 disk.drbd1777.in‚ 1.03515625 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 326.99609375 disk.rbd2.in‚0$ memory_percent‚97.93086051940918 disk.sda3.in‚7.3125 disk.rbd1.out‚ 3.12939453125 disk.nvme0c0n1.in‚5.3125Þmk’€‚(:„ ­ÕäÔ„µ­x gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;~Æg‚(:Ü ­ÕäÔ«‹Ìz disk.rbd1.in‚0& proc.memory.availableMB‚ 339.171875) disk./.usagePercent‚73.71495891754935% disk./.usageGB‚1098.4061584472656 disk.drbd1777.in‚ 1.03515625 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚327.625 disk.rbd0.out‚0$ memory_percent‚97.92985916137695 disk.rbd1.out‚ 3.12939453125 disk.dm-1.in‚7.3125 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 327.6640625 disk.sda3.in‚7.3125 disk.rbd2.in‚0 disk.rbd2.out‚ 198.333984375 disk.rbd4.in‚0( disk.nvme0c0n1.out‚28245.64111328125 disk.rbd3.in‚0& disk.nvme0n1.out‚28245.64111328125 disk.rbd0.in‚0 cpu‚25.288008190807286# disk.dm-0.out‚28245.64111328125 proc.memory.rssMB‚ 2861.78125 proc.cpu.threads‚17 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 network.sent‚ 1.5659435e+07 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 network.recv‚ 2.194828e+06 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.out‚327.625 disk.dm-0.in‚5.3125) proc.memory.percent‚17.46692657470703…ÃRš‚‚(: ¼ÕäÔºó…x gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.096' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.819200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œœþilƒ‚(:á ¼ÕäÔþˆ¬z! disk.rbd2.out‚198.50634765625 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.sda2.in‚0) proc.memory.percent‚17.46692657470703 disk.rbd4.out‚0 disk.sda1.out‚0# disk.dm-0.out‚28344.75830078125 proc.cpu.threads‚17$ disk./.usageGB‚1098.405673980713 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.out‚0 disk.nvme0n1.in‚5.3125 disk.dm-1.out‚ 328.15625 cpu‚25.28748671247685 network.sent‚ 1.5664764e+07( disk./.usagePercent‚73.7149264045925 disk.sda3.out‚ 328.15625 disk.sda2.out‚ 0.0390625 proc.memory.rssMB‚ 2861.78125 network.recv‚ 2.198161e+06 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125$ memory_percent‚97.92983531951904 disk.rbd0.in‚0 disk.sda3.in‚7.3125& disk.nvme0n1.out‚28344.75830078125 disk.dm-0.in‚5.3125( proc.memory.availableMB‚ 339.17578125( disk.nvme0c0n1.out‚28344.75830078125 disk.rbd3.in‚0 disk.sda.out‚ 328.1953125 disk.rbd1.out‚ 3.12939453125 disk.dm-1.in‚7.3125”( 42„- _wandbruntime‚6328  _runtime‚6328G„A#Æ…À & lr-AdamW/pg1‚0.0004127979982327125 & lr-AdamW/pg2‚0.0004127979982327125 trainer/global_step‚6549 $ _timestamp‚1.788422857891787e+09 _runtime‚6328.641721172 _step‚535Š(_ŠË†Å & lr-AdamW/pg1‚0.0004127979982327125 & lr-AdamW/pg2‚0.0004127979982327125 trainer/global_step‚6549 $ _timestamp‚1.788422857891787e+09 _runtime‚6328.641721172 _step‚535—à_’~2‡- _wandbruntime‚6328  _runtime‚6328'Z$Jí'ˆçO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291204759 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114723879 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44064355 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12161723769344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120776443 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120776443 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚271526425012736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚259364947449856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚271526425012736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290686092099584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12126422510830 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12374637315072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚271526671219200 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120776445 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚290686386958336 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12161723769344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚245591498449920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259364929446912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170428314 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170402644 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 _step‚536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12126198861682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 % _timestamp‚1.7884228578925915e+09 _runtime‚6328.642779075 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚271526671219200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291179087 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚6549 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚278311725703168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259364929446912 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0•]¹ò'‰ìO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120776443 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120776443 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚271526425012736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚259364947449856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚271526425012736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290686092099584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12126422510830 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12374637315072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚271526671219200 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120776445 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚290686386958336 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12161723769344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚245591498449920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259364929446912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170428314 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170402644 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 _step‚536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12126198861682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 % _timestamp‚1.7884228578925915e+09 _runtime‚6328.642779075 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚271526671219200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291179087 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚6549 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚278311725703168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259364929446912 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291204759 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114723879 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44064355 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12161723769344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2˜² ¾2Š- _wandbruntime‚6329  _runtime‚6329Õˆõº‹ñM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚278352919339520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _runtime‚6329.516191447 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70670535 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12376229682688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚259403285393920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12163333260288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12128254910638 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120794164 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚271566846871040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291222357 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44070513 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚245627783868416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170453891 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚290729125082112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚245627800907776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _step‚537 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291222357 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291248055 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚271566600651264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚259403267390976 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚259403267390976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120794164 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290729426219008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120794162 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114740400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114741050 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44069865 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170428195 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚259403285393920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12163333260288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12128254910638 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120794164 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚271566846871040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291222357 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44070513 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚245627783868416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170453891 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚290729125082112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚245627800907776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _step‚537 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291222357 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291248055 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚271566600651264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚259403267390976 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚259403267390976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120794164 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290729426219008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120794162 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114740400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114741050 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44069865 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170428195 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12467705519616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121662393 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚273535467559424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚261281844653056 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚261281826650112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚261281826650112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12253622906368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44389340 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 286470144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171683546 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121662395 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115567924 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71177930 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 293345939 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121662395 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚261281844653056 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚654 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 262530048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚247406621342720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12253622906368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚273535467559424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚292837916035584 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273535221352960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚280370234456064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12253394702848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12218056082038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _runtime‚6374.674150071 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 293371611 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259624436736250 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171709216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12467968049664 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273535221352960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚540 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171709216 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171683546 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚259624677424758 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 trainer/global_step‚6599 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12217832432890 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71177930 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 293345939 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121662395 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚261281844653056 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚654 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 262530048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚247406621342720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12253622906368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚273535467559424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚292837916035584 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273535221352960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚280370234456064 LDDeviceSta¥‡§ztsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12253394702848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12218056082038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _runtime‚6374.674150071 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 293371611 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259624436736250 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171709216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12467968049664 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273535221352960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚540 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171709216 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171683546 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚259624677424758 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 trainer/global_step‚6599 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12217832432890 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12467705519616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121662393 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚273535467559424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚261281844653056 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚261281826650112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚261281826650112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12253622906368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44389340 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 286470144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880œBõ5å‹œ‚(:€ øÕäÔú—x gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝîÊrn‚(:ã øÕäÔ€Æz) disk./.usagePercent‚73.71507616900004% disk./.usageGB‚1098.4079055786133 disk.rbd4.out‚0 disk.sda.out‚ 330.37109375 disk.sda3.out‚ 330.33203125 disk.drbd1777.out‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125! disk.rbd2.out‚200.28466796875 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 proc.memory.rssMB‚ 2861.78125& disk.nvme0c0n1.out‚28643.919921875 disk.sda3.in‚7.3125 disk.drbd1777.in‚ 1.03515625 proc.cpu.threads‚17 disk.dm-1.out‚ 330.33203125 network.sent‚ 1.5791007e+07 network.recv‚ 2.217133e+06 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.rbd4.in‚0! disk.dm-0.out‚28643.919921875' proc.memory.availableMB‚ 338.7578125) proc.memory.percent‚17.46692657470703 disk.rbd3.in‚0$ disk.nvme0n1.out‚28643.919921875 disk.dm-0.in‚5.3125 disk.rbd0.out‚0$ memory_percent‚97.93238639831543 disk.nvme0c0n1.in‚5.3125 disk.rbd1.out‚ 3.12939453125 disk.rbd2.in‚0 cpu‚25.28525911316893àÈ&ð2ž- _wandbruntime‚6375  _runtime‚63751Dnî÷&ŸñM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293414907 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12219664832490 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12255460614144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚247442906761216 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171734793 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12255460614144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121680112 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171734793 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259662812282422 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171709097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12219888481846 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12255232397312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121680112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273575396991488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71188287 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚280411380212224 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292881262257664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚261320182597120 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121680114 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 trainer/global_step‚6599 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273575643211264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚261320164594176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293389209 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44396404 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12469858105344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121680114 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚259662571593706 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115584691 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115584044 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293414907 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44395759 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884229047932646e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293389209 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292880963217920 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚6375.543677323 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121680112 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171734793 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259662812282422 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171709097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12219888481846 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12255232397312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121680112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273575396991488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71188287 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚280411380212224 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292881262257664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚261320182597120 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121680114 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 trainer/global_step‚6599 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273575643211264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚261320164594176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293389209 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44396404 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12469858105344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121680114 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚259662571593706 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115584691 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115584044 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293414907 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44395759 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884229047932646e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293389209 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292880963217920 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚6375.543677323 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293414907 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12219664832490 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673Ã92¡- _wandbruntime‚6375  _runtime‚6375”‰¬]¢× % _timestamp‚1.7884229047949114e+09 0loss/FRt3/assignment_loss‚3.1510903000831605 /loss/FRt2/detection_loss‚1.4560787912796844 0loss/SRqqt1/detection_loss‚2.230410607629701 0loss/SRqqt2/detection_loss‚0.878955599654353 /loss/FBt1/detection_loss‚2.8936186177389964 %loss/total_loss‚43.18223901208537 _step‚542 0loss/FRt1/assignment_loss‚3.6432305788382506 2loss/SRqqt4/assignment_loss‚1.7626999070698566 /loss/FRt1/detection_loss‚2.6263198053989654 1loss/SRqqt3/detection_loss‚1.7847216765759355 epoch‚5 0loss/FRt4/assignment_loss‚3.0410449166555664 /loss/FBt3/detection_loss‚2.3903858831950595 2loss/SRqqt2/assignment_loss‚1.0775442195378921 1loss/SRqqt4/detection_loss‚2.2242440351031045 _runtime‚6375.544714869 2loss/SRqqt1/assignment_loss‚2.0083362256225787 0loss/FBt1/assignment_loss‚0.9492699916507783 1loss/FBt4/assignment_loss‚0.21084422926004562 .loss/FBt2/detection_loss‚0.440798838368871 1loss/FBt2/assignment_loss‚0.10574828541061257 .loss/FBt4/detection_loss‚2.202935152583652 2loss/SRqqt3/assignment_loss‚1.5819357653795663 0loss/FBt3/assignment_loss‚0.0751127737457864 /loss/FRt4/detection_loss‚2.4493957371325106 0loss/FRt2/assignment_loss‚1.3299196752634916 /loss/FRt3/detection_loss‚2.6675973989069464 trainer/global_step‚6599°šjÚb£Ü 0loss/FRt2/assignment_loss‚1.3299196752634916 /loss/FRt3/detection_loss‚2.6675973989069464 trainer/global_step‚6599 % _timestamp‚1.7884229047949114e+09 0loss/FRt3/assignment_loss‚3.1510903000831605 /loss/FRt2/detection_loss‚1.4560787912796844 0loss/SRqqt1/detection_loss‚2.230410607629701 0loss/SRqqt2/detection_loss‚0.878955599654353 /loss/FBt1/detection_loss‚2.8936186177389964 %loss/total_loss‚43.18223901208537 _step‚542 0loss/FRt1/assignment_loss‚3.6432305788382506 2loss/SRqqt4/assignment_loss‚1.7626999070698566 /loss/FRt1/detection_loss‚2.6263198053989654 1loss/SRqqt3/detection_loss‚1.7847216765759355 epoch‚5 0loss/FRt4/assignment_loss‚3.0410449166555664 /loss/FBt3/detection_loss‚2.3903858831950595 2loss/SRqqt2/assignment_loss‚1.0775442195378921 1loss/SRqqt4/detection_loss‚2.2242440351031045 _runtime‚6375.544714869 2loss/SRqqt1/assignment_loss‚2.0083362256225787 0loss/FBt1/assignment_loss‚0.9492699916507783 1loss/FBt4/assignment_loss‚0.21084422926004562 .loss/FBt2/detection_loss‚0.440798838368871 1loss/FBt2/assignment_loss‚0.10574828541061257 .loss/FBt4/detection_loss‚2.202935152583652 2loss/SRqqt3/assignment_loss‚1.5819357653795663 0loss/FBt3/assignment_loss‚0.0751127737457864 /loss/FRt4/detection_loss‚2.4493957371325106ž8ŸÖŒ›¤‚(: ‡ÖäÔœºx gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚103.261' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.652199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Ø¥‚(:ã ‡ÖäÔ…À½z disk.rbd3.out‚0& disk.nvme0n1.out‚28743.09423828125$ memory_percent‚97.93245792388916( proc.memory.avaiT€^ßlableMB‚ 338.74609375 proc.memory.rssMB‚ 2861.78125 network.sent‚ 1.588827e+07) disk./.usagePercent‚73.71507488896236$ disk./.usageGB‚1098.407886505127 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.rbd2.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚28743.09423828125 disk.sda.out‚ 330.78125 disk.sda2.in‚0 network.recv‚ 2.223531e+06 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 330.7421875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚28743.09423828125 cpu‚25.28467073967659 disk.dm-0.in‚5.3125 disk.rbd4.out‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 330.7421875 disk.rbd1.in‚0 disk.drbd1777.out‚0) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17! disk.rbd2.out‚200.28466796875 disk.rbd3.in‚0 disk.rbd1.out‚ 3.12939453125 disk.dm-1.in‚7.3125þ¿4 ¦‚(:… –ÖäԻ̙x gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.447' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4894 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ™®”i§‚(:Þ –ÖäÔò·œ{ disk.nvme0n1.in‚5.3125 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625 disk.rbd3.out‚0! disk.dm-0.out‚28842.302734375 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125$ memory_percent‚97.93233871459961) disk./.usagePercent‚73.71507540097744 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 331.265625 disk.rbd4.in‚0 disk.rbd4.out‚0& disk.nvme0c0n1.out‚28842.302734375 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.12939453125 cpu‚25.28411738801856% disk./.usageGB‚1098.4078941345215! disk.rbd2.out‚200.36279296875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125& proc.memory.availableMB‚ 338.765625 network.sent‚ 1.5986199e+07 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 331.2265625 disk.nvme0c0n1.in‚5.3125 disk.sda3.out‚ 331.2265625 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 network.recv‚ 2.228986e+06$ disk.nvme0n1.out‚28842.302734375 disk.rbd0.out‚0&ßöa¨‚(:„ ¥ÖäÔÔ«›x gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ý¨>u©‚(:ê ¥ÖäÔàñ­{ disk.sda1.out‚0 cpu‚25.283610261476444 proc.cpu.threads‚17 network.sent‚ 1.5992499e+07) disk./.usagePercent‚73.71507412093976 disk.sda3.in‚7.3125! disk.rbd2.out‚201.87451171875 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0 network.recv‚ 2.233654e+06 disk.sda.out‚ 331.80859375& disk.nvme0n1.out‚28843.03662109375) proc.memory.percent‚17.46692657470703 disk.rbd0.in‚0 disk.rbd3.in‚0% disk./.usageGB‚1098.4078750610352 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0$ memory_percent‚97.93229103088379 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda.in‚ 7.36328125 disk.dm-0.in‚5.3125 disk.sda2.out‚ 0.0390625 proc.memory.rssMB‚ 2861.78125 disk.sda3.out‚ 331.76953125 disk.rbd1.out‚ 3.12939453125 disk.dm-1.in‚7.3125 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125' proc.memory.availableMB‚ 338.7734375# disk.dm-0.out‚28843.03662109375 disk.dm-1.out‚ 331.76953125 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚28843.03662109375œcF2ª- _wandbruntime‚6420  _runtime‚6420¸xGÉ«Ã ' lr-AdamW/pg1‚0.00040121724744930594 ' lr-AdamW/pg2‚0.00040121724744930594 trainer/global_step‚6649 % _timestamp‚1.7884229500403743e+09 _runtime‚6420.790299763 _step‚543áM4oÎ¬È ' lr-AdamW/pg1‚0.00040121724744930594 ' lr-AdamW/pg2‚0.00040121724744930594 trainer/global_step‚6649 % _timestamp‚1.7884229500403743e+09 _runtime‚6420.790299763 _step‚543Ÿg¢,2­- _wandbruntime‚6420  _runtime‚6420»áŽí'®çO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44713429 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44714098 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚261531193200258 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275544263899648 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122548345 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263198723853312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12345293839872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116409780 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚294989746844160 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122548343 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚261531433888766 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275544017693184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249221727196160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172964448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275544263899648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71695682 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12561324531200 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12561051515392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12345522043392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚671 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282428719268864 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275544017693184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12309689653246 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12345293839872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295538463 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116409109 yüò'¯ìO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚6420.791727364 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122548345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71695680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295512791 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263198741856256 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12345522043392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172964448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263198741856256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249221744235520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12309466004098 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚294990043800064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282428695328768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249221727196160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172964448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275544263899648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71695682 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12561324531200 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12561051515392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12345522043392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚671 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282428719268864 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275544017693184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12309689653246 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12345293839872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295538463 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116409109 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44713429 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44714098 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚261531193200258 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275544263899648 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122548345 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263198723853312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12345293839872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116409780 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚294989746844160 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122548343 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚261531433888766 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275544017693184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71706035 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12347131534336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282469888965120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295581759 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116426278 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12311522053054 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295556061 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295033077719552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚6649 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275584439551488 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122566062 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122566062 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173015695 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884229509056137e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚282469865025024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12311298403698 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275584193331712 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122566064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44720895 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _runtime‚6421.656594769 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_ÄN‰À: pool.current‚ 273002496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295581759 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12347131534336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71706037 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12562915751936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249258046693376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173015695 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12347359751168 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122566064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249258029654016 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172989999 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832Ì‹”¥ü&²öM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172989999 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _step‚545 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275584439551488 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚263237079800320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295032780776960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122566064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44720895 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _runtime‚6421.656594769 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295581759 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12347131534336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71706037 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12562915751936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249258046693376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173015695 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12347359751168 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122566064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249258029654016 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172989999 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71706035 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12347131534336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282469888965120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295581759 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116426278 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12311522053054 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295556061 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295033077719552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚6649 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275584439551488 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122566062 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122566062 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173015695 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884229509056137e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚282469865025024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12311298403698 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275584193331712 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 Boí'ÀçO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284487180141568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263438190352774 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297679643 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72213430 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12437421180416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45038095 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251036850088960 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174245350 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297142017156608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 trainer/global_step‚6699 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174271020 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117251527 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12654533767680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277552814033408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263437949664266 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12401323224454 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12401099575306 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚277553060239872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚277553060239872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚673 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884229960619452e+09 _step‚548 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72213432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45037424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12654813074944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 _runtime‚6466.812075038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174271020 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117250854 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12437192976896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128|ÁìO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174271020 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117251527 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12654533767680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277552814033408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263437949664266 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12401323224454 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12401099575306 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚277553060239872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚277553060239872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚673 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884229960619452e+09 _step‚548 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72213432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45037424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12654813074944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 _runtime‚6466.812075038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174271020 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117250854 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12437192976896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284487180141568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263438190352774 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297679643 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72213430 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12437421180416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45038095 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251036850088960 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174245350 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297142017156608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 trainer/global_step‚6699¤9vT2Â- _wandbruntime‚6467  _runtime‚6467Êä¨Ì÷&ÃñM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚265153959000576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12439030671360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45044327 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284528349837824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72223787 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12402931974906 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297185047372800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292748288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12403155624262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297748611 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174270901 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚6699 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265153977003520 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284528373777920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263476325210438 _runtime‚6467.674326152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123452014 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123452012 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 297722913 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123452014 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174296597 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297748611 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277592989671936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 % _timestamp‚1.7884229969236984e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117268112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251073152546816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265153977003520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12656404786688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123452012 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561bº;ü&ÄöM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12439258888192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277592989671936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 % _timestamp‚1.7884229969236984e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117268112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251073152546816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265153977003520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12656404786688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123452012 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚265153959000576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12439030671360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45044327 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284528349837824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72223787 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12402931974906 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297185047372800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292748288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12403155624262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297748611 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174270901 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚6699 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265153977003520 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284528373777920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263476325210438 _runtime‚6467.674326152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123452014 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123452012 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 297722913 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123452014 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174296597 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297748611 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299872167 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267032536262656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279561610373632 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267032536262656 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12529320317440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265344706128274 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265344946816782 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚252851972981760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45360708 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286545664954368 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12492733146514 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12529092113920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299293564969984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12492956795662 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚267032518259712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚267032518259712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299872167 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267032536262656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279561610373632 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267032536262656 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12529320317440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265344706128274 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265344946816782 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚252851972981760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45360708 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286545664954368 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12492733146514 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12529092113920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299293564969984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12492956795662 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚267032518259712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚267032518259712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 trainer/global_step‚6749 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚299336902672384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚299336591049728 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124337962 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45367510 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚267070856203776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12750044081664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118109711 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175551803 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265382840985730 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚279602032231936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124337964 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45368174 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299889765 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124337964 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12749756399104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚265383081674446 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚252888292478976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚279601786012160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299915463 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175577499 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118109045 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12494789195470 % _timestamp‚1.7884230434525528e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286586858590720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72741537 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124337962 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚286586834650624 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚6514.203181607 _step‚553 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299889765 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118109045 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12494789195470 % _timestamp‚1.7884230434525528e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286586858590720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72741537 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124337962 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚286586834650624 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚6514.203181607 _step‚553 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299889765 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 trainer/global_step‚6749 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚299336902672384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚299336591049728 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124337962 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45367510 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚267070856203776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12750044081664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118109711 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175551803 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265382840985730 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚279602032231936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124337964 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45368174 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299889765 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124337964 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12749756399104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚265383081674446 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚252888292478976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚279601786012160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299915463 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175577499©x~Ž2×- _wandbruntime‚6514  _runtime‚6514׈ô [ØÕ .loss/FBt3/detection_loss‚3.829481542110443 % _timestamp‚1.7884230434543386e+09 0loss/FRt2/assignment_loss‚1.4945267226014818 2loss/SRqqt3/assignment_loss‚1.5920914123681458 1loss/FBt2/assignment_loss‚0.11266617383154472 1loss/FBt3/assignment_loss‚0.31616417365148664 /loss/FRt2/detection_loss‚1.6308264283995544 1loss/SRqqt2/detection_loss‚0.8738096637100401 1loss/SRqqt3/detection_loss‚1.8378054557198829 epoch‚5 _runtime‚6514.204203252 0loss/FRt3/assignment_loss‚2.8941732197999954 /loss/FRt4/assignment_loss‚3.457334339618683 2loss/SRqqt1/assignment_loss‚1.8002585404834082 trainer/global_step‚6749 2loss/SRqqt2/assignment_loss‚1.0254881087900467 2loss/SRqqt4/assignment_loss‚1.8370066277864503 /loss/FBt1/assignment_loss‚0.403710583275938 0loss/FBt4/assignment_loss‚0.2614803210502335 1loss/SRqqt4/detection_loss‚1.9102280779582699 /loss/FRt1/assignment_loss‚4.235166817903519 /loss/FRt1/detection_loss‚2.4250661358237267 /loss/FRt3/detection_loss‚2.5891755312681197 1loss/SRqqt1/detection_loss‚1.9900036551231561 /loss/FBt1/detection_loss‚2.7259680554270744 -loss/FBt4/detection_loss‚2.47533189256986 %loss/total_loss‚44.86750355804496 _step‚554 /loss/FRt4/detection_loss‚2.3756310328467447 /loss/FBt2/detection_loss‚0.7741090459271501<×½Õ`ÙÚ 1loss/SRqqt4/detection_loss‚1.9102280779582699 /loss/FRt1/assignment_loss‚4.235166817903519 /loss/FRt1/detection_loss‚2.4250661358237267 /loss/FRt3/detection_loss‚2.5891755312681197 1loss/SRqqt1/detection_loss‚1.9900036551231561 /loss/FBt1/detection_loss‚2.7259680554270744 -loss/FBt4/detection_loss‚2.47533189256986 %loss/total_loss‚44.86750355804496 _step‚554 /loss/FRt4/detection_loss‚2.3756310328467447 /loss/FBt2/detection_loss‚0.7741090459271501 .loss/FBt3/detection_loss‚3.829481542110443 % _timestamp‚1.7884230434543386e+09 0loss/FRt2/assignment_loss‚1.4945267226014818 2loss/SRqqt3/assignment_loss‚1.5920914123681458 1loss/FBt2/assignment_loss‚0.11266617383154472 1loss/FBt3/assignment_loss‚0.31616417365148664 /loss/FRt2/detection_loss‚1.6308264283995544 1loss/SRqqt2/detection_loss‚0.8738096637100401 1loss/SRqqt3/detection_loss‚1.8378054557198829 epoch‚5 _runtime‚6514.204203252 0loss/FRt3/assignment_loss‚2.8941732197999954 /loss/FRt4/assignment_loss‚3.457334339618683 2loss/SRqqt1/assignment_loss‚1.8002585404834082 trainer/global_step‚6749 2loss/SRqqt2/assignment_loss‚1.0254881087900467 2loss/SRqqt4/assignment_loss‚1.8370066277864503 /loss/FBt1/assignment_loss‚0.403710583275938 0loss/FBt4/assignment_loss‚0.2614803210502335ª>]EÚ‚(:… Ž×äÔ¸þµx gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S˜=ApÛ‚(:å Ž×äÔ…Úz disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.sda.out‚ 335.5546875% disk./.usageGB‚1098.4081764221191 disk.dm-1.in‚7.3125 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.rbd0.out‚0 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.sda3.out‚ 335.515625& disk.nvme0n1.out‚29440.30322265625' proc.memory.availableMB‚ 338.3984375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.12939453125# disk.dm-0.out‚29440.30322265625 disk.sda2.in‚0) proc.memory.percent‚17.46692657470703 network.recv‚ 2.265601e+06 disk.rbd2.in‚0! disk.rbd2.out‚203.85986328125 disk.dm-0.in‚5.3125( disk.nvme0c0n1.out‚29440.30322265625 disk.nvme0n1.in‚5.3125 disk.dm-1.out‚ 335.515625 disk.sda1.out‚0 disk.sda2.out‚ 0.0390625$ memory_percent‚97.93457984924316 cpu‚25.279871515378762 disk.sda1.in‚ 0.05078125 proc.memory.rssMB‚ 2861.78125 network.sent‚ 1.6237526e+07) disk./.usagePercent‚73.71509434553496 disk.drbd1777.in‚ 1.03515625n‡LjÜ‚(:… ×äÔÐΗx gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÊõ¦gÝ‚(:Ü ×äÔÝŽÈ{$ memory_percent‚97.93319702148438 proc.memory.rssMB‚ 2861.78125 network.recv‚ 2.272654e+06" disk.dm-0.out‚29539.7236328125 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.rbd3.out‚0% disk.nvme0n1.out‚29539.7236328125 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125 disk.rbd1.in‚0 disk.rbd4.out‚0' disk.nvme0c0n1.out‚29539.7236328125 disk.rbd3.in‚0 disk.rbd1.out‚ 3.21533203125 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 cpu‚25.279339547825163 proc.cpu.threads‚17% disk./.usageGB‚1097.4238243103027 disk.dm-1.out‚ 336.03125 disk.sda.in‚ 7.36328125 disk.sda.out‚ 336.0703125! disk.rbd2.out‚203.85986328125 disk.sda3.out‚ 336.03125# proc.memory.availableMB‚338.625) proc.memory.percent‚17.46692657470703 network.sent‚ 1.6346518e+07) disk./.usagePercent‚73.64903364938453 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0ec+Þ‚(:„ ¬×äÔ׳x gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vmÍqß‚(:æ ¬×äÔ•¼Øz network.sent‚ 1.6351831e+07 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚29706.1494140625 disk.dm-1.out‚ 336.69921875 network.recv‚ 2.276058e+06 disk.rbd2.in‚0 disk.sda.out‚ 336.73828125 disk.rbd1.out‚ 3.27783203125 disk.nvme0n1.in‚5.3125) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17% disk.nvme0n1.out‚29706.1494140625 disk.sda2.in‚0 disk.dm-1.in‚7.3125) disk./.usagePercent‚73.64382722416069 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 336.69921875" disk.dm-0.out‚29706.1494140625 disk.sda2.out‚ 0.0390625% disk./.usageGB‚1097.3462448120117 disk.rbd2.out‚204.0126953125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 cpu‚25.278861753767018 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.dm-0.in‚5.3125# memory_percent‚97.9332685470581 proc.memory.rssMB‚ 2861.78125 disk.rbd4.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 338.61328125êf Ã2à- _wandbruntime‚6559  _runtime‚6559Hq°ßÉáà trainer/global_step‚6799 % _timestamp‚1.7884230887087095e+09 _runtime‚6559.458661435 _step‚555 ' lr-AdamW/pg1‚0.00038384904760099163 ' lr-AdamW/pg2‚0.00038384904760099163~{ÎâÈ _step‚555 ' lr-AdamW/pg1‚0.00038384904760099163 ' lr-AdamW/pg2‚0.00038384904760099163 trainer/global_step‚6799 % _timestamp‚1.7884230887087095e+09 _runtime‚6559.458661435«ž¹‰2ã- _wandbruntime‚6559  _runtime‚6559wÚã‰í'äçO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73248930 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302039019 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚288604149767168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12620991250944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176832824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚281570652920320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚556 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚254667112913920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 % _timestamp‚1.7884230887095926e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45686493 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12841326485504 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚267251703280790 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚281570652920320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281570406713856 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268949433465856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176807154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12584590366870 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125206193 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12841597404160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176832824 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12621219454464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45685816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281570406713856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚268949415462912 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125206193 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125206195 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118935425 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12584366717722 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125206195 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 trainer/global_step‚6799 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176807154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73248932 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚254667095874560 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚268949415462912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268949433465856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0k ¹B’ åìO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12620991250944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176832824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚281570652920320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚556 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚254667112913920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 % _timestamp‚1.7884230887095926e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45686493 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12841326485504 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚267251703280790 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚281570652920320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281570406713856 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GAîmFë`DeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268949433465856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176807154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12584590366870 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125206193 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12841597404160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176832824 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12621219454464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45685816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281570406713856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚268949415462912 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125206193 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125206195 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118935425 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12584366717722 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125206195 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 trainer/global_step‚6799 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176807154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73248932 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚254667095874560 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚268949415462912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268949433465856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73248930 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302039019 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚288604149767168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397¬r A‡2æ- _wandbruntime‚6560  _runtime‚65603íä÷&çñM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚268987771409920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12623057162240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚288645319463424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73259285 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176832705 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302082315 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚267289838138454 % _timestamp‚1.7884230895801697e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302056617 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176858401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281610828572160 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12843195833856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚254703415371776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125223912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12586422766678 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12622828945408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125223914 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45692789 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚557 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281610828572160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚301488515297280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚281610582352384 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176858401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12586199117322 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118951423 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12843462544896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125223912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚6799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302082315 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45692138 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299èpgðü&èöM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281610828572160 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12843195833856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚254703415371776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125223912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12586422766678 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12622828945408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125223914 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45692789 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚557 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281610828572160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚301488515297280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚281610582352384 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176858401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12586199117322 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118951423 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12843462544896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125223912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚6799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302082315 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45692138 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚268987771409920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12623057162240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚288645319463424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73259285 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176832705 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302082315 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚267289838138454 % _timestamp‚1.7884230895801697e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302056617 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176858401­— Ì<2é- _wandbruntime‚6560  _runtime‚6560Ê«iˆXêÒ /loss/FRt1/detection_loss‚2.7998838673035302 .loss/FRt4/detection_loss‚2.200471459032713 -loss/FBt3/detection_loss‚2.81261245906353 %loss/total_loss‚42.75463650652732 /loss/FRt3/assignment_loss‚2.997761288514504 2loss/SRqqt4/assignment_loss‚1.5363618943919526 1loss/SRqqt3/detection_loss‚1.7275974066169173 2loss/SRqqt1/assignment_loss‚2.0085278425146553 /loss/FRt2/detection_loss‚1.8069218849373418 /loss/FRt3/detection_loss‚2.4766995310783386 /loss/FRt1/assignment_loss‚3.347901713848114 1loss/FBt1/assignment_loss‚0.07136099517265393 1loss/SRqqt4/detection_loss‚1.9691119904213763 /loss/FBt2/detection_loss‚0.6730142592561322 /loss/FBt4/detection_loss‚2.0610359658797583 trainer/global_step‚6799 1loss/FBt3/assignment_loss‚0.16463770205155015 _runtime‚6560.331692653 0loss/FBt4/assignment_loss‚0.2172846390773581 epoch‚5 _step‚558 .loss/FBt1/detection_loss‚3.041765756905079 /loss/FRt2/assignment_loss‚1.474768856237101 2loss/SRqqt2/assignment_loss‚0.9845516942989301 1loss/SRqqt3/assignment_loss‚1.465494226595318 0loss/FBt2/assignment_loss‚0.1758854066460899 1loss/SRqqt1/detection_loss‚2.6595854382304585 /loss/FRt4/assignment_loss‚3.144054491519928 1loss/SRqqt2/detection_loss‚0.9373457369339817 % _timestamp‚1.7884230895818634e+09qÀÎú]ë× 0loss/FBt4/assignment_loss‚0.2172846390773581 epoch‚5 _step‚558 .loss/FBt1/detection_loss‚3.041765756905079 /loss/FRt2/assignment_loss‚1.474768856237101 2loss/SRqqt2/assignment_loss‚0.9845516942989301 1loss/SRqqt3/assignment_loss‚1.465494226595318 0loss/FBt2/assignment_loss‚0.1758854066460899 1loss/SRqqt1/detection_loss‚2.6595854382304585 /loss/FRt4/assignment_loss‚3.144054491519928 1loss/SRqqt2/detection_loss‚0.9373457369339817 % _timestamp‚1.7884230895818634e+09 /loss/FRt1/detection_loss‚2.7998838673035302 .loss/FRt4/detection_loss‚2.200471459032713 -loss/FBt3/detection_loss‚2.81261245906353 %loss/total_loss‚42.75463650652732 /loss/FRt3/assignment_loss‚2.997761288514504 2loss/SRqqt4/assignment_loss‚1.5363618943919526 1loss/SRqqt3/detection_loss‚1.7275974066169173 2loss/SRqqt1/assignment_loss‚2.0085278425146553 /loss/FRt2/detection_loss‚1.8069218849373418 /loss/FRt3/detection_loss‚2.4766995310783386 /loss/FRt1/assignment_loss‚3.347901713848114 1loss/FBt1/assignment_loss‚0.07136099517265393 1loss/SRqqt4/detection_loss‚1.9691119904213763 /loss/FBt2/detection_loss‚0.6730142592561322 /loss/FBt4/detection_loss‚2.0610359658797583 trainer/global_step‚6799 1loss/FBt3/assignment_loss‚0.16463770205155015 _runtime‚6560.331692653®ºø-Uì‚(:„ »×äÔ‰ê£x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚101.075' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.215 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gI:oí‚(:ä »×äÔ›ÍÒz disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda.in‚ 7.36328125 disk.sda2.out‚ 0.0390625$ memory_percent‚97.93345928192139( proc.memory.availableMB‚ 338.58203125 disk.dm-1.out‚ 337.10546875 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.nvme0n1.in‚5.3125% disk.nvme0n1.out‚29706.6552734375 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd4.in‚0 network.recv‚ 2.28059e+06( disk./.usagePercent‚73.6438400245374$ disk./.usageGB‚1097.346435546875 disk.dm-1.in‚7.3125' disk.nvme0c0n1.out‚29706.6552734375 disk.sda.out‚ 337.14453125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 337.10546875 disk.rbd4.out‚0 network.sent‚ 1.6357973e+07 disk.rbd2.out‚204.0283203125 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.27783203125 disk.dm-0.in‚5.3125" disk.dm-0.out‚29706.6552734375) proc.memory.percent‚17.46692657470703 cpu‚25.278303764431154 disk.sda3.in‚7.3125 proc.memory.rssMB‚ 2861.78125“™›î‚(: Ê×äÔÎÚ¼x gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$´Hroï‚(:ä Ê×äÔŠÕÎz' disk.nvme0c0n1.out‚29706.9755859375 disk.sda1.out‚0& proc.memory.availableMB‚ 338.328125 proc.memory.rssMB‚ 2861.78125 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda.out‚ 337.69140625 disk.sda2.out‚ 0.0390625 proc.cpu.threads‚17 disk.nvme0c0n1.in‚5.3125 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 3.31005859375 disk.dm-0.in‚5.3125) disk./.usagePercent‚73.64384616871821" disk.dm-0.out‚29706.9755859375 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 337.65234375 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚5.3125$ memory_percent‚97.93500900268555% disk./.usageGB‚1097.3465270996094 disk.dm-1.out‚ 337.65234375 disk.rbd2.out‚205.7275390625 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.out‚0% disk.nvme0n1.out‚29706.9755859375 disk.drbd1777.out‚0 disk.rbd1.in‚0) proc.memory.percent‚17.46692657470703 cpu‚25.277786054566846 network.sent‚ 1.646644e+07 network.recv‚ 2.286727e+06 disk.dm-1.in‚7.3125´ú¥8ð‚(:… Ù×äԛЧx gpu.0.gpu‚17.0 gpu.0.me?7ÆÖ2ÊôÄ # _timestamp‚1.78842313447871e+09 _runtime‚6605.228612738 _step‚559 & lr-AdamW/pg1‚0.0003780652542466574 & lr-AdamW/pg2‚0.0003780652542466574 trainer/global_step‚6849¯²2Þ•2õ- _wandbruntime‚6605  _runtime‚6605Õu~ì'öæO RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119779266 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290662658520064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚269158459744798 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12676223938078 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46012584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178088056 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283579203054080 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12713118591488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12676000288930 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126092143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚269158219056290 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178113726 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119778601 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178113726 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚665 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290662634579968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12934703915520 trainer/global_step‚6849 _step‚560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12934979028480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283579203054080 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283579449260544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126092145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚270866312666112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12713118591488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚270866330669056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚270866312666112ÌÙ ñ'÷ëO _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12676223938078 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46012584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178088056 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283579203054080 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12713118591488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12676000288930 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126092143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚269158219056290 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178113726 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119778601 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178113726 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚665 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290662634579968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12934703915520 trainer/global_step‚6849 _step‚560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12934979028480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283579203054080 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283579449260544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126092145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚270866312666112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12713118591488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚270866330669056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚270866312666112 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119779266 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290662658520064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚269158459744798 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561°@Ú2Ý2ø- _wandbruntime‚6606  _runtime‚6606÷/P<÷&ùñM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283619624912384 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126109862 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12678056337886 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73777035 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚303640373924352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚283619378692608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178139303 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 trainer/global_step‚6849 % _timestamp‚1.7884231353339806e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 _step‚561 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚270904650610176 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178139303 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126109864 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126109864 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119796704 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119796024 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304249167 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46019667 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚270904650610176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256518521225216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12714956299264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256518538264576 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12936569648128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚270904668613120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304223469 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178113607 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12714728082432 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚270904668613120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178113607 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6606.084267949 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12677832688530 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73777037 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283619624912384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12714728082432 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126109862 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46018989 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12936842650624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚269196353913746 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299n%vòúöM _runtime‚6606.084267949 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12677832688530 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73777037 ?5DeviceStatsMonitor.on_tréÂÙ &ain_batch_end/active.all.peak‚31397 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283619624912384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12714728082432 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126109862 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46018989 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12936842650624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚269196353913746 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283619624912384 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126109862 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12678056337886 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73777035 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚303640373924352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚283619378692608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178139303 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 trainer/global_step‚6849 % _timestamp‚1.7884231353339806e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 _step‚561 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚270904650610176 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178139303 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126109864 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126109864 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119796704 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119796024 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304249167 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46019667 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚270904650610176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256518521225216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12714956299264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256518538264576 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12936569648128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚270904668613120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304223469 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178113607 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12714728082432 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚270904668613120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178113607 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0±4­«—2û- _wandbruntime‚6606  _runtime‚6606%}ësYüÓ /loss/FRt1/assignment_loss‚3.181085873294521 /loss/FRt4/detection_loss‚2.3224241620785486 1loss/SRqqt3/detection_loss‚1.7036051194546586 _step‚562 2loss/SRqqt4/assignment_loss‚1.3653055252852264 /loss/FRt3/detection_loss‚2.4325737885453482 %loss/total_loss‚42.98192613268593 epoch‚5 .loss/FBt2/detection_loss‚0.485437775689983 trainer/global_step‚6849 0loss/FRt2/assignment_loss‚1.3152778264014953 1loss/SRqqt1/assignment_loss‚1.751836738452829 0loss/FBt1/assignment_loss‚0.4525368724328776 0loss/SRqqt2/detection_loss‚0.860138127929531 /loss/FBt1/detection_loss‚3.2045019467671714 /loss/FBt3/detection_loss‚3.2840867212840488 2loss/SRqqt2/assignment_loss‚0.9889269344130298 1loss/FBt3/assignment_loss‚0.27183181737616124 /loss/FRt1/detection_loss‚2.7295362377488934 .loss/FBt4/detection_loss‚2.717973694205284 0loss/SRqqt3/assignment_loss‚1.60595908085421 % _timestamp‚1.7884231353355696e+09 0loss/FRt3/assignment_loss‚2.4182139282876793 0loss/FRt4/assignment_loss‚3.5191998385094307 /loss/FRt2/detection_loss‚1.4937409089217264 0loss/SRqqt1/detection_loss‚2.681256776225978 1loss/SRqqt4/detection_loss‚1.9389769103791978 _runtime‚ 6606.08538299 1loss/FBt2/assignment_loss‚0.18159760525456226 1loss/FBt4/assignment_loss‚0.07590192289353581éu8Š^ýØ % _timestamp‚1.7884231353355696e+09 0loss/FRt3/assignment_loss‚2.4182139282876793 0loss/FRt4/assignment_loss‚3.5191998385094307 /loss/FRt2/detection_loss‚1.4937409089217264 0loss/SRqqt1/detection_loss‚2.681256776225978 1loss/SRqqt4/detection_loss‚1.9389769103791978 _runtime‚ 6606.08538299 1loss/FBt2/assignment_loss‚0.18159760525456226 1loss/FBt4/assignment_loss‚0.07590192289353581 /loss/FRt1/assignment_loss‚3.181085873294521 /loss/FRt4/detection_loss‚2.3224241620785486 1loss/SRqqt3/detection_loss‚1.7036051194546586 _step‚562 2loss/SRqqt4/assignment_loss‚1.3653055252852264 /loss/FRt3/detection_loss‚2.4325737885453482 %loss/total_loss‚42.98192613268593 epoch‚5 .loss/FBt2/detection_loss‚0.485437775689983 trainer/global_step‚6849 0loss/FRt2/assignment_loss‚1.3152778264014953 1loss/SRqqt1/assignment_loss‚1.751836738452829 0loss/FBt1/assignment_loss‚0.4525368724328776 0loss/SRqqt2/detection_loss‚0.860138127929531 /loss/FBt1/detection_loss‚3.2045019467671714 /loss/FBt3/detection_loss‚3.2840867212840488 2loss/SRqqt2/assignment_loss‚0.9889269344130298 1loss/FBt3/assignment_loss‚0.27183181737616124 /loss/FRt1/detection_loss‚2.7295362377488934 .loss/FBt4/detection_loss‚2.717973694205284 0loss/SRqqt3/assignment_loss‚1.60595908085421²vLmŽþ‚(:ƒ è×äÔŒ¹žy gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mÿ&Çkÿ‚(:à è×äÔåò§|% disk.nvme0n1.out‚29708.2626953125 disk.rbd0.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 338.51953125 cpu‚25.276781383662733% disk./.usageGB‚1097.3458557128906 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625' disk.nvme0c0n1.out‚29708.2626953125 disk.sda3.out‚ 339.4453125) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 disk.dm-1.out‚ 339.4453125 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 3.373046875" disk.dm-0.out‚29708.2626953125 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.rbd2.out‚205.8056640625 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 network.recv‚ 2.294705e+06 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.sda3.in‚7.3125$ memory_percent‚97.93384075164795 proc.memory.rssMB‚ 2861.78125( disk./.usagePercent‚73.6438011113922 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 network.sent‚ 1.6478049e+07 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 339.484375Ús뎀‚(:ƒ ÷×äÔ¢¨x gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.29' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.058 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËqìÆe‚(:Ú ÷×äÔ©¢Ô{ disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125$ disk.nvme0c0n1.out‚ 29712.1171875 disk.sda.out‚ 341.30078125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 341.26171875 proc.cpu.threads‚17 disk.dm-0.out‚ 29712.1171875 disk.drbd1777.out‚0 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 proc.memory.rssMB‚ 2861.78125 cpu‚25.2762755616068) disk./.usagePercent‚73.64384207259768 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625" disk.nvme0n1.out‚ 29712.1171875) proc.memory.percent‚17.46692657470703 disk.sda2.in‚0 disk.dm-1.out‚ 341.26171875 disk.rbd2.out‚205.8212890625 disk.rbd0.in‚0 disk.rbd4.in‚0( proc.memory.availableMB‚ 338.26171875% disk./.usageGB‚1097.3464660644531 disk.dm-0.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.nvme0n1.in‚5.3125 network.sent‚ 1.6586512e+07 network.recv‚ 2.300955e+06 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.rbd1.out‚ 3.373046875$ memory_percent‚97.93541431427002‡a ‚‚(:… †ØäÔïá´x gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.361' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š¿¦DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚564 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179394628 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚271065216208806 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74284432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚292721119392768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚292721143332864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12804789524992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272783209869312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 % _timestamp‚1.7884231803520033e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179394628 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46336470 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12767633860138 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _runtime‚ 6651.10212139 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚305749472080896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258297358699520 trainer/global_step‚6899 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12767857509286 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179368958 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272783209869312 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272783227872256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚271064975520298 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚285588245600768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚258297341660160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚285587999394304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13028051537920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2Í«À%؉ëO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272783227872256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚271064975520298 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚285588245600768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚258297341660160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚285587999394304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13028051537920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126978093 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚285588245600768 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120621563 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179368958 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272783227872256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46337131 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13028328748032 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126978095 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120620900 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12805017728512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74284430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126978095 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126978093 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚285587999394304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12804789524992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚564 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179394628 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚271065216208806 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74284432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚292721119392768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚292721143332864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12804789524992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272783209869312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 % _timestamp‚1.7884231803520033e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179394628 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46336470 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12767633860138 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _runtime‚ 6651.10212139 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚305749472080896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258297358699520 trainer/global_step‚6899 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12767857509286 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179368958 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272783209869312´â2Š- _wandbruntime‚6651  _runtime‚6651;•Qö&‹ðM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272821565816320 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚292762313029120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126995812 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚271103110377754 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120637520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306416019 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12769466259738 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272821547813376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚285628421252608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120638133 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13030190885376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74294785 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272821565816320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46342735 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚305792209069056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚292762289089024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126995814 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179420205 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13029919980032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126995812 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179420205 $ _timestamp‚1.788423181221298e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12806627219456 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126995814 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46343346 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258333644118016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 jû&ŒõM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272821565816320 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚292762313029120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126995812 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚271103110377754 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120637520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306416019 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12769466259738 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272821547813376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚285628421252608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120638133 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13030190885376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74294785 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272821565816320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46342735 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚305792209069056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚292762289089024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126995814 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179420205 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13029919980032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126995812 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179420205 $ _timestamp‚1.788423181221298e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12806627219456 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126995814 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46343346 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258333644118016µ/â¶2- _wandbruntime‚6651  _runtime‚6651…¼Úk\ŽÖ 1loss/SRqqt3/detection_loss‚1.8113782959324973 .loss/FBt3/detection_loss‚4.864675968885422 /loss/FBt4/detection_loss‚3.3916581571102142 trainer/global_step‚6899 0loss/FRt1/assignment_loss‚3.6317968777247835 /loss/FRt3/assignment_loss‚3.165500188982764 1loss/FBt4/assignment_loss‚0.04665056732483208 &loss/total_loss‚47.119748752963474 2loss/SRqqt1/assignment_loss‚1.8121363541909628 0loss/FBt3/assignment_loss‚0.7392157912254333 _step‚566 2loss/SRqqt3/assignment_loss‚1.6330245168702233 .loss/FBt1/detection_loss‚2.803878494671413 # _timestamp‚1.78842318122345e+09 0loss/FRt2/assignment_loss‚1.4453103950149135 .loss/FRt2/detection_loss‚2.232327067146176 epoch‚5 1loss/FBt2/assignment_loss‚0.12670843399217302 .loss/FRt3/detection_loss‚2.509319628393927 /loss/FRt4/detection_loss‚2.5140163045790467 1loss/SRqqt4/detection_loss‚1.9000255951285363 1loss/SRqqt2/detection_loss‚1.0576700090684674 /loss/FBt2/detection_loss‚0.6294697611855055 /loss/FRt1/detection_loss‚2.6370268753596715 2loss/SRqqt2/assignment_loss‚0.8309301249682903 2loss/SRqqt4/assignment_loss‚1.7377053274214267 1loss/FBt1/assignment_loss‚0.25773551740816664 _runtime‚6651.973252895 0loss/FRt4/assignment_loss‚3.2005049056477017 1loss/SRqqt1/detection_loss‚2.1410835947309224 ‚‰aÛ /loss/FRt4/detection_loss‚2.5140163045790467 1loss/SRqqt4/detection_loss‚1.9000255951285363 1loss/SRqqt2/detection_loss‚1.0576700090684674 /loss/FBt2/detection_loss‚0.6294697611855055 /loss/FRt1/detection_loss‚2.6370268753596715 2loss/SRqqt2/assignment_loss‚0.8309301249682903 2loss/SRqqt4/assignment_loss‚1.7377053274214267 1loss/FBt1/assignment_loss‚0.25773551740816664 _runtime‚6651.973252895 0loss/FRt4/assignment_loss‚3.2005049056477017 1loss/SRqqt1/detection_loss‚2.1410835947309224 1loss/SRqqt3/detection_loss‚1.8113782959324973 .loss/FBt3/detection_loss‚4.864675968885422 /loss/FBt4/detection_loss‚3.3916581571102142 trainer/global_step‚6899 0loss/FRt1/assignment_loss‚3.6317968777247835 /loss/FRt3/assignment_loss‚3.165500188982764 1loss/FBt4/assignment_loss‚0.04665056732483208 &loss/total_loss‚47.119748752963474 2loss/SRqqt1/assignment_loss‚1.8121363541909628 0loss/FBt3/assignment_loss‚0.7392157912254333 _step‚566 2loss/SRqqt3/assignment_loss‚1.6330245168702233 .loss/FBt1/detection_loss‚2.803878494671413 # _timestamp‚1.78842318122345e+09 0loss/FRt2/assignment_loss‚1.4453103950149135 .loss/FRt2/detection_loss‚2.232327067146176 epoch‚5 1loss/FBt2/assignment_loss‚0.12670843399217302 .loss/FRt3/detection_loss‚2.509319628393927¶µ“áXš‚(: •ØäÔலx gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚90.667' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.133399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž^f¢t‘‚(:é •ØäÔ¦¢¢{) disk./.usagePercent‚73.63237779920686 disk.nvme0n1.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda.in‚ 7.36328125$ memory_percent‚97.93751239776611 disk.rbd4.in‚0 disk.rbd3.in‚0 proc.memory.rssMB‚ 2861.78125 network.recv‚ 2.30987e+06% disk./.usageGB‚1097.1756401062012 disk.sda3.in‚7.3125 disk.rbd1.out‚ 3.43603515625 disk.rbd0.out‚0 disk.rbd1.in‚0) proc.memory.percent‚17.46692657470703& disk.nvme0n1.out‚29716.32373046875 disk.sda2.in‚0 disk.dm-1.in‚7.3125 cpu‚25.27526325972254 disk.dm-0.in‚5.3125# disk.dm-0.out‚29716.32373046875 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.sda.out‚ 343.39453125 disk.rbd2.in‚0! disk.rbd2.out‚205.92724609375 disk.dm-1.out‚ 343.35546875 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚29716.32373046875 disk.sda1.out‚0 disk.sda3.out‚ 343.35546875( proc.memory.availableMB‚ 337.91796875 network.sent‚ 1.6598811e+07 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 proc.cpu.threads‚17ˆ!)’‚(:… ¤ØäÔ½’Žx gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.494' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.6988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jý~&s“‚(:è ¤ØäÔºƒ´z! disk.rbd2.out‚207.56396484375 disk.drbd1777.in‚ 1.03515625$ disk./.usageGB‚1097.175636291504 disk.rbd1.in‚0& disk.nvme0n1.out‚29716.66357421875 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 3.43603515625 disk.dm-1.out‚ 344.28515625 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125$ memory_percent‚97.93601036071777 network.recv‚ 2.315503e+06 disk.rbd2.in‚0( disk.nvme0c0n1.out‚29716.66357421875 disk.sda1.out‚0) proc.memory.percent‚17.46692657470703 network.sent‚ 1.6706694e+07 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.drbd1777.out‚0 disk.rbd4.out‚0' proc.memory.availableMB‚ 338.1640625 cpu‚25.27476586112892) disk./.usagePercent‚73.63237754319933 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 344.32421875# disk.dm-0.out‚29716.66357421875 disk.sda3.out‚ 344.28515625 proc.memory.rssMB‚ 2861.78125Êj¿¹”‚(:‚ ³ØäÔÑÍ€x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.86' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷•õ}Ü•‚(:è ³ØäÔÏëé{ disk.sda3.out‚ 344.55078125 disk.nvme0n1.in‚5.3125 network.sent‚ 1.6712698e+07 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 network.recv‚ 2.319714e+06 disk.dm-0.in‚5.3125 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 cpu‚25.274178360625733( disk.nvme0c0n1.out‚29850.45654296875 disk.rbd1.in‚0 disk.rbd1.out‚ 3.43603515625& disk.nvme0n1.out‚29850.45654296875) disk./.usagePercent‚73.64110484004158 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125& proc.memory.availableMB‚ 338.390625 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125! disk.rbd2.out‚207.57958984375 disk.drbd1777.out‚0 disk.sda1.out‚0$ memory_percent‚97.93462753295898 proc.cpu.threads‚17# disk.dm-0.out‚29850.45654296875 disk.dm-1.out‚ 344.55078125 disk.sda.in‚ 7.36328125 disk.sda.zzZ»—out‚ 344.58984375 disk.sda3.in‚7.3125$ disk./.usageGB‚1097.305679321289 disk.rbd2.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0/“Ý32–- _wandbruntime‚6697  _runtime‚6697¾„&7Æ—À & lr-AdamW/pg1‚0.0003665133101846555 & lr-AdamW/pg2‚0.0003665133101846555 trainer/global_step‚6949 $ _timestamp‚1.788423226504632e+09 _runtime‚6697.254575936 _step‚567”ºû•Ë˜Å & lr-AdamW/pg1‚0.0003665133101846555 & lr-AdamW/pg2‚0.0003665133101846555 trainer/global_step‚6949 $ _timestamp‚1.788423226504632e+09 _runtime‚6697.254575936 _step‚567·Ê?Pˆ2™- _wandbruntime‚6697  _runtime‚6697¥Èì'šæO UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127864045 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180675530 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180675530 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274700125075456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127864043 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12896916865536 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127864043 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚287597041940992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12896688662016 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121462959 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13121682622464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 $ _timestamp‚1.788423226505391e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74802182 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127864045 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180649860 _step‚568 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚287596795734528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46661457 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46660779 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180649860 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274700107072512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308539575 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308513903 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12896916865536 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272971731984306 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12859491080494 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74802180 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚294779604205568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13121409606656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚287597041940992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307901310768128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 trainer/global_step‚6949 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚287596795734528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272971972672814 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚260112481592320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12896688662016 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12859267431346 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262÷岚ñ'›ëO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180649860 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274700107072512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308539575 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308513903 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12896916865536 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272971731984306 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12859491080494 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74802180 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚294779604205568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13121409606656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚287597041940992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307901310768128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 trainer/global_step‚6949 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚287596795734528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272971972672814 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚260112481592320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12896688662016 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12859267431346 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127864045 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180675530 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180675530 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274700125075456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127864043 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12896916865536 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127864043 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚287597041940992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12896688662016 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121462959 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13121682622464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 $ _timestamp‚1.788423226505391e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74802182 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127864045 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180649860 _step‚568 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚287596795734528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46661457 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46660779 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896¸æËÛ»2œ- _wandbruntime‚6698  _runtime‚6698vYõ{÷&ñM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307944362459136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12898754573312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308582871 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚287637217592832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 308557173 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚294820797841920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚260148767010816 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12861099830946 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121480249 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚273009866841762 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127881764 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚287637217592832 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274738463019520 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12898526356480 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274738445016576 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274738445016576 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12898526356480 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13123564617216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚287636971373056 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13123287420416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚569 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 trainer/global_step‚6949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74812535 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 308582871 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127881762 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180675411 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46668364 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚294820773901824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308582871 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚287637217592832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 308557173 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚294820797841920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚260148767010816 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12861099830946 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121480249 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚273009866841762 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127881764 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚287637217592832 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274738463019520 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12898526356480 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274738445016576 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274738445016576 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12898526356480 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13123564617216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚287636971373056 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13123287420416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚569 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 trainer/global_step‚6949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74812535 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 308582871 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127881762 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180675411 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46668364 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚294820773901824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 Pc§‚(:Ø àØäÔ§¨ { disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 cpu‚25.272653777604194 network.sent‚ 1.6832774e+07 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125( disk.nvme0c0n1.out‚29985.00634765625 network.recv‚ 2.333931e+06% disk./.usageGB‚1097.4362297058105 disk.sda3.out‚ 346.203125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 346.2421875! memory_percent‚97.93701171875 proc.memory.rssMB‚ 2861.78125) disk./.usagePercent‚73.64986618588588 disk.drbd1777.out‚0& disk.nvme0n1.out‚29985.00634765625 disk.rbd4.out‚0 disk.rbd1.out‚ 3.43603515625# disk.dm-0.out‚29985.00634765625 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 proc.memory.availableMB‚338 disk.rbd2.out‚ 207.806640625 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 346.203125) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 disk.sda.in‚ 7.36328125 disk.dm-0.in‚5.3125 disk.nvme0c0n1.in‚5.3125†ŒH2¨- _wandbruntime‚6744  _runtime‚6744`(È©Â _runtime‚6744.875517902 _step‚571 ' lr-AdamW/pg1‚0.00036074756688481586 ' lr-AdamW/pg2‚0.00036074756688481586 trainer/global_step‚6999 $ _timestamp‚1.788423274125584e+09òä”껂 _runtime‚6744.875517902 _step‚571 ' lr-AdamW/pg1‚0.00036074756688481586 ' lr-AdamW/pg2‚0.00036074756688481586 trainer/global_step‚6999 $ _timestamp‚1.788423274125584e+09»òç2«- _wandbruntime‚6744  _runtime‚6744ªEJ‘ë'¬åO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46986648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274878729136822 # _timestamp‚1.78842327412635e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12988587799040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296838089018368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181930762 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75319930 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13215079408128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _runtime‚6744.876474114 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128749995 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276617004275712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276617004275712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75319932 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12988816002560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289605592074752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚261927587445760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181930762 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122305892 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181956432 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289605838281216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181956432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚310052897507840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 310706427 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 trainer/global_step‚6999 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12988587799040 _step‚572 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13214808489472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289605592074752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128749995 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289605838281216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚256729ò¶Ïç­êO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12951124651702 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128749993 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12988816002560 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289605838281216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181956432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚310052897507840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 310706427 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 trainer/global_step‚6999 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12988587799040 _step‚572 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13214808489472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289605592074752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128749995 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289605838281216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46986648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274878729136822 # _timestamp‚1.78842327412635e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12988587799040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296838089018368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181930762 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75319930 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13215079408128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6ì 893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _runtime‚6744.876474114 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128749995 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276617004275712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276617004275712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75319932 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12988816002560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289605592074752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚261927587445760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181930762 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122305892 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181956432 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276655342219776 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122322448 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13216953713152 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75330287 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚310095931036672 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276655342219776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 _runtime‚6745.751333076 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310749723 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276655360222720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 trainer/global_step‚6999 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310724025 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296879282654720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289646013933056 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚310096236367872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128767712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12952957051510 _step‚573 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12990425493504 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12990653710336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274916863994486 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181982009 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310749723 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122323105 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚261963889903616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ±vü&°öM PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122323105 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚261963889903616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289645767713280 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181956313 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128767714 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚261963906942976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296879258714624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276655342219776 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122322448 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13216953713152 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75330287 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚310095931036672 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276655342219776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 _runtime‚6745.751333076 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310749723 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276655360222720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 trainer/global_step‚6999 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310724025 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296879282654720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289646013933056 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚310096236367872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128767712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12952957051510 _step‚573 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12990425493504 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12990653710336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274916863994486 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181982009 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310749723 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824½&ìì2±- _wandbruntime‚6745  _runtime‚6745~:Þó[²Õ 0loss/FBt1/assignment_loss‚0.6470258260020879 /loss/FRt3/detection_loss‚2.4975266996771097 /loss/SRqqt1/detection_loss‚2.36653200785319 1loss/SRqqt4/detection_loss‚1.8576224555774612 /loss/FBt2/detection_loss‚0.6432305556941735 2loss/SRqqt4/assignment_loss‚1.4714433209476423 1loss/SRqqt2/detection_loss‚1.1201240588330998 .loss/FBt1/detection_loss‚2.197548766930898 /loss/FRt3/assignment_loss‚3.435312253733476 2loss/SRqqt1/assignment_loss‚1.9665878893751088 1loss/SRqqt3/detection_loss‚1.9167478091042975 /loss/FBt4/detection_loss‚2.5331395169099173 epoch‚5 % _timestamp‚1.7884232750025969e+09 /loss/FRt2/assignment_loss‚1.314516686476194 0loss/FRt4/assignment_loss‚3.2854193108422414 _runtime‚6745.752439351 trainer/global_step‚6999 2loss/SRqqt3/assignment_loss‚1.6129159778356552 1loss/FBt3/assignment_loss‚0.34282969304695143 /loss/FBt4/assignment_loss‚0.559860012649248 0loss/FRt1/assignment_loss‚3.6649150295955377 2loss/SRqqt2/assignment_loss‚0.9328934721437827 /loss/FRt4/detection_loss‚2.1566760326482157 _step‚574 /loss/FRt2/detection_loss‚1.8265336327827895 /loss/FBt3/detection_loss‚3.0537514686584473 1loss/FBt2/assignment_loss‚0.08721079347370692 /loss/FRt1/detection_loss‚2.3779408204846266 %loss/total_loss‚43.86830409127586÷nÁ`³Ú 2loss/SRqqt3/assignment_loss‚1.6129159778356552 1loss/FBt3/assignment_loss‚0.34282969304695143 /loss/FBt4/assignment_loss‚0.559860012649248 0loss/FRt1/assignment_loss‚3.6649150295955377 2loss/SRqqt2/assignment_loss‚0.9328934721437827 /loss/FRt4/detection_loss‚2.1566760326482157 _step‚574 /loss/FRt2/detection_loss‚1.8265336327827895 /loss/FBt3/detection_loss‚3.0537514686584473 1loss/FBt2/assignment_loss‚0.08721079347370692 /loss/FRt1/detection_loss‚2.3779408204846266 %loss/total_loss‚43.86830409127586 0loss/FBt1/assignment_loss‚0.6470258260020879 /loss/FRt3/detection_loss‚2.4975266996771097 /loss/SRqqt1/detection_loss‚2.36653200785319 1loss/SRqqt4/detection_loss‚1.8576224555774612 /loss/FBt2/detection_loss‚0.6432305556941735 2loss/SRqqt4/assignment_loss‚1.4714433209476423 1loss/SRqqt2/detection_loss‚1.1201240588330998 .loss/FBt1/detection_loss‚2.197548766930898 /loss/FRt3/assignment_loss‚3.435312253733476 2loss/SRqqt1/assignment_loss‚1.9665878893751088 1loss/SRqqt3/detection_loss‚1.9167478091042975 /loss/FBt4/detection_loss‚2.5331395169099173 epoch‚5 % _timestamp‚1.7884232750025969e+09 /loss/FRt2/assignment_loss‚1.314516686476194 0loss/FRt4/assignment_loss‚3.2854193108422414 _runtime‚6745.752439351 trainer/global_step‚6999¾ûQ´‚(:„ ïØäÔ…å©x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.235' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.647 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„cRw`µ‚(:Õ ïØäÔ±™çz disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.sda2.out‚ 0.0390625 proc.cpu.threads‚17% disk./.usageGB‚1097.4443969726562 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125$ memory_percent‚97.93834686279297) proc.memory.percent‚17.46692657470703 network.sent‚ 1.6838895e+07 disk.nvme0c0n1.in‚5.3125" disk.nvme0n1.out‚ 29985.4609375 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 29985.4609375% proc.memory.availableMB‚ 337.78125 disk.sda3.out‚ 346.7578125 disk.rbd1.out‚ 3.43603515625 disk.rbd2.out‚ 209.396484375 proc.memory.rssMB‚ 2861.78125 disk.dm-0.out‚ 29985.4609375) disk./.usagePercent‚73.65041429801668 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.rbd2.in‚0 network.recv‚ 2.33832e+06 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda.out‚ 346.796875 cpu‚25.272127527559682 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 346.7578125 disk.sda1.in‚ 0.05078125ˆG¹¶‚(:… þØäÔ…úýw gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.872' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K…EÜg·‚(:Ü þØäԱѷ{ disk.sda1.out‚0" disk.nvme0n1.out‚ 29994.0234375 disk.rbd2.out‚ 209.458984375 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.43603515625 disk.dm-1.out‚ 347.12109375$ memory_percent‚97.93775081634521 network.sent‚ 1.6947291e+07) disk./.usagePercent‚73.65040047360985 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 347.12109375 disk.nvme0n1.in‚5.3125 network.recv‚ 2.344397e+06 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.dm-0.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125$ disk.nvme0c0n1.out‚ 29994.0234375 disk.sda.out‚ 347.16015625 proc.memory.rssMB‚ 2861.78125$ disk./.usageGB‚1097.444190979004 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125) proc.memory.percent‚17.46692657470703 cpu‚25.271632151796247 disk.rbd0.out‚0 disk.dm-0.out‚ 29994.0234375 disk.drbd1777.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 337.87890625 proc.cpu.threads‚17q_¸‚(:„ ÙäÔûþ›x gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@jDnq¹‚(:æ ÙäÔ›éêz disk.dm-1.out‚ 347.77734375 network.sent‚ 1.695361e+07 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚29994.47705078125 disk.dm-0.in‚5.3125 disk.rbd2.out‚ 209.474609375 network.recv‚ 2.349092e+06 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 3.43603515625# disk.dm-0.out‚29994.47705078125 cpu‚25.271148272278367) disk./.usagePercent‚73.65039919357217 disk.nvme0c0n1.in‚5.3125 disk.sda.in‚ 7.36328125 proc.cpu.threads‚17% disk./.usageGB‚1097.4441719055176 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚ 347.81640625 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0( disk.nvme0c0n1.out‚29994.47705078125 disk.sda2.in‚0 disk.rbd2.in‚0# memory_percent‚97.9377269744873 disk.rbd3.out‚0 disk.dm-1.in‚7.3125' proc.memory.availableMB‚ 337.8828125 disk.drbd1777.out‚0 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 347.77734375 disk.rbd1.in‚0ÀmS2º- _wandbruntime‚6791  _runtime‚6791~Å€$ɻà % _timestamp‚1.7884233204358058e+09 _runtime‚6791.185722325 _step‚575 ' lr-AdamW/pg1‚0.00035499024533699086 ' lr-AdamW/pg2‚0.00035499024533699086 trainer/global_step‚7049âd`EÎ¼È % _timestamp‚1.7884233204358058e+09 _runtime‚6791.185722325 _step‚575 ' lr-AdamW/pg1‚0.00035499024533699086 ' lr-AdamW/pg2‚0.00035499024533699086 trainer/global_step‚7049¿éI2½- _wandbruntime‚6791  _runtime‚6791_ëµó†¾çO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚291614634621440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13042534573762 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 WIDeviceStatsMon²çÂ)g%itor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13080486936064 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13080486936064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312873279 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47316105 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚278533901478912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298896573831168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚276785485600830 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129635945 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129635943 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183237334 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183237334 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183211664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884233204365287e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123153787 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚291614388414976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75837682 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75837680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚276785244912322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚291614634621440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚278533919481856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129635945 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚312204858106368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312847607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129635943 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 trainer/global_step‚7049 _step‚576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚278533919481856 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183211664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚278533901478912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13308284275200 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13308563582464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚263742710338560 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13042758222910 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13080715139584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚291614388414976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298896597771264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13080715139584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0Î!ßò'¿ìO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚291614634621440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13042534573762 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13080486936064 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13080486936064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312873279 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47316105 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚278533901478912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298896573831168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚276785485600830 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129635945 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129635943 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183237334 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183237334 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183211664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884233204365287e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123153787 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚291614388414976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75837682 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75837680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚276785244912322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚291614634621440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚278533919481856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129635945 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚312204858106368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312847607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129635943 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 trainer/global_step‚7049 _step‚576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚278533919481856 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183211664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚278533901478912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13308284275200 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13308563582464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚263742710338560 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13042758222910 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13080715139584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚291614388414976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298896597771264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13080715139584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0Àêð6|2À- _wandbruntime‚6792  _runtime‚6792g|Zö&ÁðM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚276823620458494 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚6792.057458674 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 305331200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13082324630528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13082324630528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123170309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183262911 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13310143312896 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚276823379769778 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚291654810273280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚278572257425920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183237215 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312916575 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚291654810273280 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13082552847360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123170968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75848037 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚298937767467520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚7049 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129653664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263779029835776 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13044590622718 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129653664 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚312247886840320 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75848035 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183237215 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312916575 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47322931 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚657 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47322274 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚577 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚278572239422976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13310424704000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13082552847360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312248192171520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚278572257425920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263779012796416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824k,;*\ ÂõM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47322274 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚577 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚278572239422976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13310424704000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13082552847360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312248192171520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚278572257425920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263779012796416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚276823620458494 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚6792.057458674 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 305331200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13082324630528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13082324630528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123170309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183262911 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13310143312896 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚276823379769778 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚291654810273280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚278572257425920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183237215 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312916575 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚291654810273280 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13082552847360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123170968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75848037 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚298937767467520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚7049 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129653664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263779029835776 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13044590622718 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129653664 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚312247886840320 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75848035 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183237215 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312916575 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47322931 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚657Áž‡¯62Ã- _wandbruntime‚6792  _runtime‚6792£Ùˆ^ÄØ 2loss/SRqqt3/assignment_loss‚1.7584229060360845 1loss/SRqqt3/detection_loss‚1.8861794595448476 1loss/FBt1/assignment_loss‚0.10565214226436284 /loss/FRt3/detection_loss‚2.5647190199341883 .loss/FBt3/detection_loss‚7.431552946567535 _runtime‚6792.058522829 .loss/FRt2/detection_loss‚1.673978440530066 2loss/SRqqt2/assignment_loss‚1.0567455349986752 2loss/SRqqt4/assignment_loss‚1.6738110920069393 epoch‚5 /loss/FRt4/assignment_loss‚3.251713718686785 2loss/FBt3/assignment_loss‚0.007153020240366459 0loss/FBt4/assignment_loss‚0.6978907673037611 1loss/SRqqt1/detection_loss‚2.1646837294101715 /loss/FBt2/detection_loss‚0.5702000848181984 /loss/FRt1/assignment_loss‚3.914067409255288 0loss/FRt2/assignment_loss‚1.1417766649808203 0loss/FRt3/assignment_loss‚3.6192522797473643 /loss/FRt1/detection_loss‚2.8372526132699214 1loss/SRqqt2/detection_loss‚0.7896163483460744 /loss/FBt1/detection_loss‚2.3635045488675437 2loss/SRqqt1/assignment_loss‚1.9252583872940805 .loss/FRt4/detection_loss‚2.340361090339014 .loss/FBt4/detection_loss‚2.660975955426693 &loss/total_loss‚48.305766683869585 trainer/global_step‚7049 1loss/FBt2/assignment_loss‚0.15226956112903928 1loss/SRqqt4/detection_loss‚1.7187289628717635 % _timestamp‚1.7884233213086784e+09 _step‚578tÑ…~cÅÝ 1loss/FBt1/assignment_loss‚0.10565214226436284 /loss/FRt3/detection_loss‚2.5647190199341883 .loss/FBt3/detection_loss‚7.431552946567535 _runtime‚6792.058522829 .loss/FRt2/detection_loss‚1.673978440530066 2loss/SRqqt2/assignment_loss‚1.0567455349986752 2loss/SRqqt4/assignment_loss‚1.6738110920069393 epoch‚5 /loss/FRt4/assignment_loss‚3.251713718686785 2loss/FBt3/assignment_loss‚0.007153020240366459 0loss/FBt4/assignment_loss‚0.6978907673037611 1loss/SRqqt1/detection_loss‚2.1646837294101715 /loss/FBt2/detection_loss‚0.5702000848181984 /loss/FRt1/assignment_loss‚3.914067409255288 0loss/FRt2/assignment_loss‚1.1417766649808203 0loss/FRt3/assignment_loss‚3.6192522797473643 /loss/FRt1/detection_loss‚2.8372526132699214 1loss/SRqqt2/detection_loss‚0.7896163483460744 /loss/FBt1/detection_loss‚2.3635045488675437 2loss/SRqqt1/assignment_loss‚1.9252583872940805 .loss/FRt4/detection_loss‚2.340361090339014 .loss/FBt4/detection_loss‚2.660975955426693 &loss/total_loss‚48.305766683869585 trainer/global_step‚7049 1loss/FBt2/assignment_loss‚0.15226956112903928 1loss/SRqqt4/detection_loss‚1.7187289628717635 % _timestamp‚1.7884233213086784e+09 _step‚578 2loss/SRqqt3/assignment_loss‚1.7584229060360845 1loss/SRqqt3/detection_loss‚1.8861794595448476Â\ úÆ‚(:… œÙäÔüí‘y gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.158' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'EmœtÇ‚(:é œÙäÔ¿¬ß{ disk.sda3.out‚ 348.63671875 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17! disk.rbd2.out‚209.56103515625 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚7.3125' proc.memory.availableMB‚ 337.9453125 network.recv‚ 2.352449e+06 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda3.in‚7.3125 disk.dm-0.in‚5.3125# disk.dm-0.out‚30102.42236328125 disk.drbd1777.in‚ 1.03515625 cpu‚25.270691569907008 disk.rbd1.out‚ 3.43603515625 network.sent‚ 1.6958871e+07( disk./.usagePercent‚73.6593453768562% disk./.usageGB‚1097.5774765014648 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0& disk.nvme0n1.out‚30102.42236328125 disk.dm-1.out‚ 348.63671875 disk.sda.in‚ 7.36328125 disk.sda.out‚ 348.67578125 disk.rbd4.in‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚30102.42236328125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚5.3125$ memory_percent‚97.93734550476074 disk.sda1.in‚ 0.05078125’fqœšÈ‚(: «ÙäÔý¥x gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.57' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.913999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½ÚpÉ‚(:å «ÙäÔŽ½áz disk.sda.out‚ 349.12890625) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 disk.sda2.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.43603515625( proc.memory.availableMB‚ 338.41796875 proc.memory.rssMB‚ 2861.78125' disk.nvme0c0n1.out‚30103.5126953125 disk.nvme0n1.in‚5.3125 cpu‚25.27017841256376 network.recv‚ 2.359189e+06 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚209.57275390625 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125 disk.sda3.out‚ 349.08984375$ memory_percent‚97.93446063995361 disk.sda1.out‚0 disk.sda.in‚ 7.36328125% disk.nvme0n1.out‚30103.5126953125 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 349.08984375 disk.rbd4.out‚0 disk.rbd3.in‚0 network.sent‚ 1.7067542e+07) disk./.usagePercent‚73.66784610703064# disk./.usageGB‚1097.70414352417 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0" disk.dm-0.out‚30103.5126953125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.drbd1777.out‚0 disk.rbd3.out‚0}[ñÊ‚(:„ ºÙäÔõšƒx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛæõkË‚(:à ºÙäÔɾ´z disk.drbd1777.out‚0 disk.sda.in‚ 7.36328125% disk./.usageGB‚1097.9546737670898) proc.memory.percent‚17.46692657470703 disk.dm-1.out‚ 349.8359375 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125$ memory_percent‚97.93529510498047 proc.cpu.threads‚17 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125# disk.dm-0.out‚30202.72119140625( disk.nvme0c0n1.out‚30202.72119140625 disk.rbd3.out‚0 network.recv‚ 2.36321e+06! disk.rbd2.out‚209.69775390625 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.out‚349.875 disk.sda3.out‚ 349.8359375 cpu‚25.269735545339557) disk./.usagePercent‚73.68465940184188 disk.rbd1.out‚ 3.43603515625 disk.sda2.out‚ 0.0390625 disk.rbd4.out‚0 disk.sda1.out‚0 disk.sda3.in‚7.3125 network.sent‚ 1.7073404e+07& disk.nvme0n1.out‚30202.72119140625 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125% proc.memory.availableMB‚ 338.28125 proc.memory.rssMB‚ 2861.78125 disk.rbd1.in‚0 disk.dm-1.in‚7.3125 disk.rbd0.in‚0°Ý:2Ì- _wandbruntime‚6837  _runtime‚6837fÑZþÆÍÀ $ _timestamp‚1.788423366600741e+09 _runtime‚6837.350669492 _step‚579 & lr-AdamW/pg1‚0.0003492425453583621 & lr-AdamW/pg2‚0.0003492425453583621 trainer/global_step‚7099¿×ìËÎÅ _step‚579 & lr-AdamW/pg1‚0.0003492425453583621 & lr-AdamW/pg2‚0.0003492425453583621 trainer/global_step‚7099 $ _timestamp‚1.788423366600741e+09 _runtime‚6837.350669492ÃcÇDp2Ï- _wandbruntime‚6837  _runtime‚6837xøPí'ÐçO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130521893 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315014459 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13172614276608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13134391794118 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184518236 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293623184755200 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280450816685056 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123996367 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13134168144970 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184492566 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278692242064838 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884233666014988e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47640937 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13401935991808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚293623184755200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚293623430961664 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130521893 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚280450816685056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚265557850270720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚265557833231360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13172386073088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47641597 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184492566 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚314356715328512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7099 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚314357018575872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 _step‚580 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130521895œánÑìO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚293623184755200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚293623430961664 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130521893 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚280450816685056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚265557850270720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚265557833231360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13172386073088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47641597 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184492566 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚314356715328512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7099 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚314357018575872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 _step‚580 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130521895 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130521893 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315014459 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13172614276608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13134391794118 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184518236 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293623184755200 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280450816685056 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123996367 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13134168144970 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184492566 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278692242064838 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884233666014988e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47640937 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13401935991808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508Ä,‚—z2Ò- _wandbruntime‚6838  _runtime‚6838`á÷&ÓñM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13174223767552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚581 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293663606613504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300996228340224 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 315057729 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280489154629120 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130539614 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚278730376922502 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47647917 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184543813 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130539612 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6838.248992307 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280489154629120 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76365787 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265594152728576 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13136000544570 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130539614 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315083427 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130539612 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13174451984384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315057729 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184518117 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124013025 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚314400052335104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚314399749101056 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13403800054784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280489136626176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293663360393728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124013704 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262ŹÏqü&ÔöM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265594152728576 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13136000544570 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130539614 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315083427 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130539612 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13174451984384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315057729 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184518117 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124013025 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚314400052335104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚314399749101056 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13403800054784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280489136626176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293663360393728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124013704 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13174223767552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚581 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293663606613504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300996228340224 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 315057729 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280489154629120 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130539614 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚278730376922502 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47647917 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184543813 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130539612 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6838.248992307 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280489154629120 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76365787Å×Cý`2Õ- _wandbruntime‚6838  _runtime‚6838”ÙŽ§YÖÓ /loss/FRt3/assignment_loss‚2.721479158103466 0loss/FBt4/assignment_loss‚0.1839787196367979 1loss/SRqqt4/detection_loss‚2.0821147948503493 0loss/FRt4/assignment_loss‚2.8501745085967216 _runtime‚6838.250035161 2loss/SRqqt1/assignment_loss‚1.8828003131426299 .loss/FRt2/detection_loss‚1.484155821055174 -loss/FRt4/detection_loss‚2.66781706715885 /loss/FBt4/detection_loss‚2.6641298797395496 trainer/global_step‚7099 /loss/FRt2/assignment_loss‚1.662148904800415 2loss/SRqqt3/assignment_loss‚1.6402794772640188 1loss/FBt3/assignment_loss‚0.19852822377288248 .loss/FRt3/detection_loss‚2.607329297810793 .loss/FBt1/detection_loss‚2.297286244539114 1loss/FBt1/assignment_loss‚0.08472913285144247 1loss/FBt2/assignment_loss‚0.08029259024341624 /loss/FRt1/detection_loss‚2.7974007134051884 1loss/SRqqt1/detection_loss‚2.3500436819516697 0loss/SRqqt3/detection_loss‚1.801201011906279 _step‚582 0loss/FRt1/assignment_loss‚3.4935363916789783 2loss/SRqqt2/assignment_loss‚0.9543370373462999 1loss/SRqqt2/detection_loss‚0.7890864507709112 .loss/FBt2/detection_loss‚0.574619568368218 /loss/FBt3/detection_loss‚2.3099260479211807 %loss/total_loss‚41.46461018661499 epoch‚5 % _timestamp‚1.7884233675002022e+09 2loss/SRqqt4/assignment_loss‚1.2872151497006417¯¨‹Ù^×Ø 0loss/FRt1/assignment_loss‚3.4935363916789783 2loss/SRqqt2/assignment_loss‚0.9543370373462999 1loss/SRqqt2/detection_loss‚0.7890864507709112 .loss/FBt2/detection_loss‚0.574619568368218 /loss/FBt3/detection_loss‚2.3099260479211807 %loss/total_loss‚41.46461018661499 epoch‚5 % _timestamp‚1.7884233675002022e+09 2loss/SRqqt4/assignment_loss‚1.2872151497006417 /loss/FRt3/assignment_loss‚2.721479158103466 0loss/FBt4/assignment_loss‚0.1839787196367979 1loss/SRqqt4/detection_loss‚2.0821147948503493 0loss/FRt4/assignment_loss‚2.8501745085967216 _runtime‚6838.250035161 2loss/SRqqt1/assignment_loss‚1.8828003131426299 .loss/FRt2/detection_loss‚1.484155821055174 -loss/FRt4/detection_loss‚2.66781706715885 /loss/FBt4/detection_loss‚2.6641298797395496 trainer/global_step‚7099 /loss/FRt2/assignment_loss‚1.662148904800415 2loss/SRqqt3/assignment_loss‚1.6402794772640188 1loss/FBt3/assignment_loss‚0.19852822377288248 .loss/FRt3/detection_loss‚2.607329297810793 .loss/FBt1/detection_loss‚2.297286244539114 1loss/FBt1/assignment_loss‚0.08472913285144247 1loss/FBt2/assignment_loss‚0.08029259024341624 /loss/FRt1/detection_loss‚2.7974007134051884 1loss/SRqqt1/detection_loss‚2.3500436819516697 0loss/SRqqt3/detection_loss‚1.801201011906279 _step‚582Æ7ËQÞØ‚(:… ÉÙäÔ³ûËx gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.322' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ãyßtÙ‚(:é ÉÙäÔŠÕüz disk.dm-0.in‚5.3125 disk.drbd1777.in‚ 1.03515625% disk./.usageGB‚1097.9536361694336 disk.nvme0c0n1.in‚5.3125) proc.memory.percent‚17.46692657470703 disk.sda3.out‚ 350.29296875 disk.rbd1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.rbd2.in‚0( disk.nvme0c0n1.out‚30302.36474609375 disk.sda3.in‚7.3125 disk.sda2.out‚ 0.0390625 proc.cpu.threads‚17 disk.rbd4.in‚0 disk.rbd3.in‚0# memory_percent‚97.9356050491333 network.sent‚ 1.707981e+07 disk.sda.out‚ 350.33203125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125( proc.memory.availableMB‚ 338.23046875 network.recv‚ 2.367969e+06) disk./.usagePercent‚73.68458976779256 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚30302.36474609375! disk.rbd2.out‚211.28759765625 disk.sda1.out‚0 disk.rbd1.out‚ 3.43603515625& disk.nvme0n1.out‚30302.36474609375 disk.dm-1.out‚ 350.29296875 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.rbd3.out‚0 proc.memory.rssMB‚ 2861.78125 cpu‚25.269214753938016·¬^Ú‚(:… ØÙäÔà³£x gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊZ±pÛ‚(:å ØÙäÔÔ¤‘| proc.cpu.threads‚17 network.recv‚ 2.37356e+06! disk.rbd2.out‚211.41259765625 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 network.sent‚ 1.7188195e+07 disk.sda.out‚ 350.9921875 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 350.953125 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚5.3125 disk.rbd4.out‚0 disk.sda3.out‚ 350.953125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.43603515625& disk.nvme0n1.out‚30401.50537109375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚30401.50537109375 cpu‚25.268764421245514 disk.sda.in‚ 7.36328125 disk.rbd4.in‚0) disk./.usagePercent‚73.68458976779256 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.46692657470703 disk.sda3.in‚7.3125 disk.dm-0.in‚5.3125# disk.dm-0.out‚30401.50537109375( proc.memory.availableMB‚ 337.99609375 proc.memory.rssMB‚ 2861.78125% disk./.usageGB‚1097.9536361694336 disk.sda2.out‚ 0.0390625$ memory_percent‚97.93703556060791&€»Ü‚(:… çÙäÔìÛ«x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/è!ÏtÝ‚(:é çÙäÔ䎕| disk.rbd1.in‚0 disk.dm-1.out‚ 351.50390625 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 proc.memory.rssMB‚ 2861.78125 disk.dm-0.in‚5.3125 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0 cpu‚25.26825223052853 disk.sda2.in‚0 disk.sda.out‚ 351.54296875 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚30402.44189453125 disk.nvme0c0n1.in‚5.3125) proc.memory.percent‚17.46692657470703 disk.rbd0.out‚0 disk.rbd4.in‚0) disk./.usagePercent‚73.68458976779256& disk.nvme0n1.out‚30402.44189453125! disk.rbd2.out‚211.42822265625 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1097.9536361694336( disk.nvme0c0n1.out‚30402.44189453125 disk.sda3.out‚ 351.50390625# memory_percent‚97.9356050491333( proc.memory.availableMB‚ 338.23046875 disk.rbd1.out‚ 3.43603515625 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd3.out‚0 disk.sda3.in‚7.3125 network.sent‚ 1.7194213e+07 network.recv‚ 2.378027e+06I;“2Þ- _wandbruntime‚6883  _runtime‚6883²Fí9Ãß½ trainer/global_step‚7149 % _timestamp‚1.7884234130156322e+09 _runtime‚6883.765549302 _step‚583 $ lr-AdamW/pg1‚0.00034350566476099 $ lr-AdamW/pg2‚0.00034350566476099ÈÃBÓÈà $ lr-AdamW/pg1‚0.00034350566476099 $ lr-AdamW/pg2‚0.00034350566476099 trainer/global_step‚7149 % _timestamp‚1.7884234130156322e+09 _runtime‚6883.765549302 _step‚583Ç0‡Z2á- _wandbruntime‚6883  _runtime‚6883AÄþŸâæO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13264285210112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13495419501056 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47965960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281404416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚280598757840338 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131407845 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚295631981095424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13225801716178 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124839142 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocateý»-M d‚299 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚295631981095424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282367713888256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 _step‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚651 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303013543456768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185799138 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131407845 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47965311 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303013567396864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚295632227301888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131407843 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124838491 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 317206983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267372956124160 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13226025365326 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚316508986897920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317181311 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13495138096640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚282367695885312 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267372973163520 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76873182 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13264285210112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚295632227301888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282367713888256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 trainer/global_step‚7149 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185773468 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282367695885312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185773468 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303013567396864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚295632227301888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131407843 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124838491 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 317206983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267372956124160 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13226025365326 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚316508986897920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317181311 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13495138096640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚282367695885312 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267372973163520 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76873182 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13264285210112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚295632227301888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282367713888256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 trainer/global_step‚7149 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185773468 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282367695885312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185773468 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131407845 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚295631981095424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13225801716178 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124839142 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚295631981095424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282367713888256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 _step‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚651 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303013543456768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824Èœ-Œú2ä- _wandbruntime‚6884  _runtime‚6884ñy£)ö&åðM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13266122904576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚295672402953728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47972812 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 317224581 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316551723867648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303054713153024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚280636892697794 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131425562 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131425564 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282406033829376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131425564 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13266351121408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124855670 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131425562 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267409258582016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317250279 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185799019 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13266122904576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76883537 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13227857765134 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚295672402953728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317250279 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282406051832320 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267409275621376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124856349 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13497281619968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13497010714624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316552018713088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47972135 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13266351121408 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185799019 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185824715 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303054737093120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13227857765134 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚295672402953728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317250279 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282406051832320 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267409275621376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124856349 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13497281619968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13497010714624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316552018713088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47972135 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13266351121408 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185799019 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185824715 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303054737093120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 317224581 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316551723867648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303054713153024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚280636892697794 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131425562 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131425564 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282406033829376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131425564 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13266351121408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124855670 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131425562 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267409258582016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317250279 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185799019 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13266122904576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76883537 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ÉèZ°2ç- _wandbruntime‚6884  _runtime‚688417/úZèÔ _runtime‚6884.641256883 0loss/FRt2/assignment_loss‚1.6983753876252607 .loss/FRt3/assignment_loss‚3.22349473263355 2loss/SRqqt1/assignment_loss‚1.6427668861367486 epoch‚5 2loss/SRqqt4/assignment_loss‚1.5248605073714743 1loss/FBt1/assignment_loss‚0.04982376552510459 /loss/FRt1/assignment_loss‚3.031620120129934 2loss/SRqqt3/assignment_loss‚1.8533981640053832 1loss/FBt2/assignment_loss‚0.14713245349210266 1loss/FBt4/assignment_loss‚0.18564171533216722 0loss/SRqqt1/detection_loss‚2.549953560034434 1loss/SRqqt3/detection_loss‚1.8498801433521768 /loss/FBt4/detection_loss‚2.1318120509386063 /loss/FRt2/detection_loss‚1.4695647828280927 /loss/FBt2/detection_loss‚0.7485961851387007 2loss/SRqqt2/assignment_loss‚0.8284185585987477 .loss/FRt1/detection_loss‚2.762700362903316 1loss/SRqqt2/detection_loss‚0.9016404332643425 -loss/FRt3/detection_loss‚2.49606373398862 _step‚586 1loss/FBt3/assignment_loss‚0.17391509550442505 .loss/FBt1/detection_loss‚2.370910309255123 /loss/FBt3/detection_loss‚2.0875812570254006 %loss/total_loss‚40.66507965385859 trainer/global_step‚7149 0loss/FRt4/assignment_loss‚2.6460405826568603 /loss/FRt4/detection_loss‚2.3658041979634015 1loss/SRqqt4/detection_loss‚1.9250846681546192 $ _timestamp‚1.788423413891444e+09ðZ¶£_éÙ 1loss/FBt3/assignment_loss‚0.17391509550442505 .loss/FBt1/detection_loss‚2.370910309255123 /loss/FBt3/detection_loss‚2.0875812570254006 %loss/total_loss‚40.66507965385859 trainer/global_step‚7149 0loss/FRt4/assignment_loss‚2.6460405826568603 /loss/FRt4/detection_loss‚2.3658041979634015 1loss/SRqqt4/detection_loss‚1.9250846681546192 $ _timestamp‚1.788423413891444e+09 _runtime‚6884.641256883 0loss/FRt2/assignment_loss‚1.6983753876252607 .loss/FRt3/assignment_loss‚3.22349473263355 2loss/SRqqt1/assignment_loss‚1.6427668861367486 epoch‚5 2loss/SRqqt4/assignment_loss‚1.5248605073714743 1loss/FBt1/assignment_loss‚0.04982376552510459 /loss/FRt1/assignment_loss‚3.031620120129934 2loss/SRqqt3/assignment_loss‚1.8533981640053832 1loss/FBt2/assignment_loss‚0.14713245349210266 1loss/FBt4/assignment_loss‚0.18564171533216722 0loss/SRqqt1/detection_loss‚2.549953560034434 1loss/SRqqt3/detection_loss‚1.8498801433521768 /loss/FBt4/detection_loss‚2.1318120509386063 /loss/FRt2/detection_loss‚1.4695647828280927 /loss/FBt2/detection_loss‚0.7485961851387007 2loss/SRqqt2/assignment_loss‚0.8284185585987477 .loss/FRt1/detection_loss‚2.762700362903316 1loss/SRqqt2/detection_loss‚0.9016404332643425 -loss/FRt3/detection_loss‚2.49606373398862 _step‚586Ê£uê‚(:… öÙäÔ „œx gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.447' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4894 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OZÙ[të‚(:é öÙäÔöºÝz disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 proc.cpu.threads‚17% disk./.usageGB‚1098.4563941955566 disk.rbd0.in‚0( disk.nvme0c0n1.out‚30501.57861328125 disk.sda3.out‚ 352.07421875) disk./.usagePercent‚73.71833028076776) proc.memory.percent‚17.46692657470703# disk.dm-0.out‚30501.57861328125 disk.sda2.in‚0 disk.rbd2.out‚211.5771484375 disk.drbd1777.in‚ 1.03515625 disk.sda3.in‚7.3125( proc.memory.availableMB‚ 337.99609375 disk.rbd1.out‚ 3.43603515625 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 proc.memory.rssMB‚ 2861.78125 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 352.07421875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 352.11328125$ memory_percent‚97.93703556060791 network.sent‚ 1.7199111e+07 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 cpu‚25.26782550585808& disk.nvme0n1.out‚30501.57861328125 disk.dm-1.in‚7.3125 network.recv‚ 2.381254e+06d³Róì‚(:„ …ÚäÔÜ”x gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.515' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.703 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%ýOJpí‚(:å …ÚäÔΞÕz disk.sda.in‚ 7.36328125% disk./.usageGB‚1098.4543380737305& disk.nvme0n1.out‚30600.76220703125 disk.sda1.out‚0) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 352.7578125 cpu‚25.267318218383558 network.sent‚ 1.7307981e+07 network.recv‚ 2.388023e+06 disk.dm-0.in‚5.3125 disk.sda2.in‚0( disk.nvme0c0n1.out‚30600.76220703125 disk.rbd1.in‚0$ memory_percent‚97.93603420257568 disk.rbd1.out‚ 3.43603515625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda3.in‚7.3125 proc.memory.rssMB‚ 2861.78125( disk./.usagePercent‚73.7181922927068 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚30600.76220703125 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125( proc.memory.availableMB‚ 338.16015625 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚211.5927734375 disk.rbd0.out‚0 disk.sda.out‚ 352.796875 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 352.7578125 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0»Þû½›î‚(: ”ÚäÔ퉾x gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊVÈ·rï‚(:ç ”ÚäÔ¸×ãz disk.drbd1777.out‚0% proc.memory.availableMB‚ 338.03125 proc.memory.rssMB‚ 2861.78125 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0$ memory_percent‚97.93682098388672) disk./.usagePercent‚73.69638710698543 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 353.44921875) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625 disk.dm-0.in‚5.3125 disk.rbd2.out‚213.2529296875 disk.rbd0.out‚0 disk.rbd4.in‚0 network.sent‚ 1.7313301e+07 disk.dm-1.out‚ 353.44921875 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125 cpu‚25.266847697567687 disk.dm-1.in‚7.3125 network.recv‚ 2.391061e+06 disk.rbd1.out‚ 3.43603515625# disk.dm-0.out‚30699.90673828125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0& disk.nvme0n1.out‚30699.90673828125 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚30699.90673828125% disk./.usageGB‚1098.1294250488281 disk.nvme0n1.in‚5.3125 disk.rbd0.in‚0 disk.sda.out‚ 353.48828125Ó×2ð- _wandbruntime‚6929  _runtime‚6929–} ¿ÇñÁ _runtime‚6929.914717958 _step‚587 & lr-AdamW/pg1‚0.0003377807991021925 & lr-AdamW/pg2‚0.0003377807991021925 trainer/global_step‚7199 % _timestamp‚1.7884234591647906e+09ƒ²hÌòÆ trainer/global_step‚7199 % _timestamp‚1.7884234591647906e+09 _runtime‚6929.914717958 _step‚587 & lr-AdamW/pg1‚0.0003377807991021925 & lr-AdamW/pg2‚0.0003377807991021925Ë|¤Š2ó- _wandbruntime‚6929  _runtime‚6929ìU¢ˆí'ôçO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48292108 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚6929.916199861 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13356412550656 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚269188079016960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 319373835 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚284284611091456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305072052209664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132293793 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13588913711104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282505514304346 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284284593088512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚7199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13356412550656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187054370 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚284284611091456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 319348163 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297640777435648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚644 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187080040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚318660965920768 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13356184347136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13588640695296 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚297641023642112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚269188096056320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚284284593088512 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13317435287386 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 % _timestamp‚1.7884234591658094e+09 _step‚588 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305072052209664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132293793 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13588913711104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282505514304346 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284284593088512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚7199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13356412550656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187054370 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚284284611091456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 319348163 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297640777435648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚644 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187080040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚318660965920768 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13356184347136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13588640695296 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚297641023642112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚269188096056320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚284284593088512 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13317435287386 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 % _timestamp‚1.7884234591658094e+09 _step‚588 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297681199293952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚284322949035520 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 319391433 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282543889850518 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77401285 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13590527766016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚6930.792151453 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132311514 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13358250258432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187105617 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884234600414538e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚318704016382976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚305113221905920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13319491336342 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚284322931032576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚284322949035520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297680953074176 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 319417131 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚269224381474816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48298475 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚284322931032576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48299107 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 trainer/global_step‚7199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125699760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 319417131 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187079921 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13358022041600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚589 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13358250258432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0);{!ü&úöM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 319391433 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884234600414538e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚318704016382976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚305113221905920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13319491336342 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚284322931032576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚284322949035520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297680953074176 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 319417131 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚269224381474816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48298475 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚284322931032576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48299107 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 trainer/global_step‚7199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125699760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 319417131 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187079921 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13358022041600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚589 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13358250258432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297681199293952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚305113197965824 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282543649161802 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13319267686986 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77401287 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297681199293952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚284322949035520 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 319391433 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282543889850518 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77401285 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13590527766016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚6930.792151453 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132311514 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13358250258432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187105617 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceSta[ˆ“£B!tsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286201508294656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚286201490291712 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188335272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13448083484160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321515015 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13448311687680 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13448083484160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 321540687 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133179745 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126526928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _runtime‚6976.133393668 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚320812697834496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271003218949120 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133179743 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚647 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299649819982336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77908682 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188335272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307130513082368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚299649573775872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299649573775872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13409068858594 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133179743 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚320813001081856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307130537022464 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286201508294656 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133179745 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13448311687680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13682464059392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271003201909760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚299649819982336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48617599 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126526279 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286201490291712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 trainer/global_step‚7249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188360942 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360¼ ²†ò'‰ìO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚299649573775872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299649573775872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13409068858594 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133179743 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚320813001081856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307130537022464 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286201508294656 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133179745 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13448311687680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13682464059392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271003201909760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚299649819982336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48617599 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126526279 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286201490291712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 trainer/global_step‚7249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188360942 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188360942 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77908680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286201508294656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚286201490291712 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188335272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13448083484160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321515015 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13448311687680 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13448083484160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 321540687 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133179745 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126526928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _runtime‚6976.133393668 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚320812697834496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271003218949120 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133179743 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚647 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299649819982336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77908682 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188335272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307130513082368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0Ð\Mý2Š- _wandbruntime‚6977  _runtime‚6977(hÕò÷&‹ñM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77919035 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13411124907550 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚299689995634176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚299689749414400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48623666 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13684056159744 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292748288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286239846238720 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13449921178624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48624309 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 321583983 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321583983 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚284450405625810 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 321558285 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188386519 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286239846238720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚320856031686656 trainer/global_step‚7249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13450149395456 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307171682778624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133197464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚307171706718720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133197462 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 321558285 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133197462 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126542701 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚299689995634176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133197464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77919037 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188360823 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126543346 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561rpXR€ŒöM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286239846238720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚320856031686656 trainer/global_step‚7249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13450149395456 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307171682778624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133197464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚307171706718720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133197462 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 321558285 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133197462 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126542701 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚299689995634176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133197464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77919037 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188360823 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126543346 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77919035 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13411124907550 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚g|299689995634176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚299689749414400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48623666 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13684056159744 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292748288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286239846238720 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13449921178624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48624309 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 321583983 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321583983 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚284450405625810 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 321558285 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188386519 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896ѧÇ'ç2- _wandbruntime‚6977  _runtime‚6977!+F4^ŽØ /loss/FBt4/detection_loss‚3.2862025002638497 /loss/FRt2/detection_loss‚1.8708797067257703 0loss/FBt4/assignment_loss‚0.3144423269368417 /loss/FBt2/detection_loss‚0.6504988010274246 trainer/global_step‚7249 0loss/FRt1/assignment_loss‚3.1863433465361597 1loss/FBt2/assignment_loss‚0.15587625263235041 epoch‚5 1loss/SRqqt3/detection_loss‚1.6854284088762979 0loss/FRt2/assignment_loss‚1.1898155411084492 1loss/SRqqt1/assignment_loss‚1.976052316526572 2loss/SRqqt2/assignment_loss‚0.9280790479913834 /loss/FRt3/detection_loss‚2.4836783580323485 0loss/FRt4/assignment_loss‚3.4259985143488105 2loss/SRqqt4/assignment_loss‚1.4031786452598718 /loss/FRt4/detection_loss‚2.3184646517039513 1loss/SRqqt2/detection_loss‚0.8860237117135717 /loss/FBt3/detection_loss‚2.9304780695173474 2loss/SRqqt3/assignment_loss‚1.3313012211805297 % _timestamp‚1.7884235062664962e+09 %loss/total_loss‚43.04250249857678 1loss/SRqqt4/detection_loss‚1.8624655634773022 .loss/FBt1/detection_loss‚2.124371713399887 .loss/FRt1/detection_loss‚2.625039319694042 /loss/FRt3/assignment_loss‚3.330409998589374 1loss/FBt1/assignment_loss‚0.26609719735570253 1loss/FBt3/assignment_loss‚0.40534474329130415 _runtime‚6977.016299154 _step‚594 1loss/SRqqt1/detection_loss‚2.4060325423876443¤~:âcÝ 0loss/FRt4/assignment_loss‚3.4259985143488105 2loss/SRqqt4/assignment_loss‚1.4031786452598718 /loss/FRt4/detection_loss‚2.3184646517039513 1loss/SRqqt2/detection_loss‚0.8860237117135717 /loss/FBt3/detection_loss‚2.9304780695173474 2loss/SRqqt3/assignment_loss‚1.3313012211805297 % _timestamp‚1.7884235062664962e+09 %loss/total_loss‚43.04250249857678 1loss/SRqqt4/detection_loss‚1.8624655634773022 .loss/FBt1/detection_loss‚2.124371713399887 .loss/FRt1/detection_loss‚2.625039319694042 /loss/FRt3/assignment_loss‚3.330409998589374 1loss/FBt1/assignment_loss‚0.26609719735570253 1loss/FBt3/assignment_loss‚0.40534474329130415 _runtime‚6977.016299154 _step‚594 1loss/SRqqt1/detection_loss‚2.4060325423876443 /loss/FBt4/detection_loss‚3.2862025002638497 /loss/FRt2/detection_loss‚1.8708797067257703 0loss/FBt4/assignment_loss‚0.3144423269368417 /loss/FBt2/detection_loss‚0.6504988010274246 trainer/global_step‚7249 0loss/FRt1/assignment_loss‚3.1863433465361597 1loss/FBt2/assignment_loss‚0.15587625263235041 epoch‚5 1loss/SRqqt3/detection_loss‚1.6854284088762979 0loss/FRt2/assignment_loss‚1.1898155411084492 1loss/SRqqt1/assignment_loss‚1.976052316526572 2loss/SRqqt2/assignment_loss‚0.9280790479913834 /loss/FRt3/detection_loss‚2.4836783580323485Òœk4øš‚(: ßÚäÔôæ£x gpu.0.gpu‚44.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.648' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.729599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GI+³p‘‚(:å ßÚäÔÆ“{ network.sent‚ 1.7525985e+07 network.recv‚ 2.415565e+06% disk./.usageGB‚1098.8780975341797 disk.nvme0n1.in‚5.3125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚30999.2626953125$ memory_percent‚97.94111251831055 proc.memory.rssMB‚ 2861.78125! disk.rbd2.out‚213.51123046875 disk.rbd4.out‚0) disk./.usagePercent‚73.74663114565543 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.rbd1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 356.15234375& proc.memory.availableMB‚ 337.328125 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 356.15234375 disk.rbd1.out‚ 3.43603515625" disk.dm-0.out‚30999.2626953125 disk.dm-1.in‚7.3125) proc.memory.percent‚17.46692657470703 cpu‚25.27090491527897 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚30999.2626953125 disk.sda.in‚ 7.36328125 disk.dm-0.in‚5.3125 disk.rbd0.in‚0 disk.sda.out‚ 356.19140625 disk.rbd4.in‚0 disk.sda2.out‚ 0.0390625 proc.cpu.threads‚17ƒ<÷õ’‚(:… îÚäÔØ…Ñz gpu.0.gpu‚44.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚124.038' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8076 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó{‡œm“‚(:â îÚäÔϤ³ network.sent‚ 1.7634466e+07 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.rbd0.in‚0! disk.rbd2.out‚215.08544921875 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 337.32421875 proc.memory.rssMB‚ 2861.78125 disk.nvme0c0n1.in‚5.3125$ memory_percent‚97.94113636016846 proc.cpu.threads‚17) disk./.usagePercent‚73.74663114565543% disk./.usageGB‚1098.8780975341797 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.rbd1.out‚ 3.43603515625 network.recv‚ 2.421771e+06 disk.drbd1777.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 356.84375) proc.memory.percent‚17.46692657470703 cpu‚25.28021470646247 disk.dm-0.in‚5.3125" disk.dm-0.out‚31098.3994140625 disk.rbd4.in‚0 disk.sda3.out‚ 356.8046875 disk.rbd1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 356.8046875' disk.nvme0c0n1.out‚31098.3994140625 disk.nvme0n1.in‚5.3125% disk.nvme0n1.out‚31098.3994140625X¦z”‚(:„ ýÚäÔô»Ÿx gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚123.797' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û;ßb•‚(:× ýÚäÔÜ×Í{ proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703) disk./.usagePercent‚73.74663856987392 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 357.1953125 network.sent‚ 1.7640761e+07 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0" disk.nvme0n1.out‚ 31197.9765625 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 357.15625 proc.cpu.threads‚17 disk.rbd4.out‚0 disk.dm-0.out‚ 31197.9765625 cpu‚25.289261495531157 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0$ disk.nvme0c0n1.out‚ 31197.9765625 disk.sda3.in‚7.3125 disk.sda2.in‚0$ memory_percent‚97.94268608093262 network.recv‚ 2.426472e+06 disk.sda3.out‚ 357.15625! disk.rbd2.out‚215.10107421875 disk.rbd0.in‚0' proc.memory.availableMB‚ 337.0703125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.43603515625 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0% disk./.usageGB‚1098.8782081604004 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚5.3125 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.sda1.out‚0±†)2–- _wandbruntime‚7033  _runtime‚7033‹ŠBS*—ÍT B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚561 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚287674078710272 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6039636992 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚262 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 322893357 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚322390669186560 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31397 ^QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 322873154 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚78399693 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚561 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚301150667287552 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20203 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 6893338624 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚285835262576442 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚23 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚272397625751552 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _step‚595 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31168 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚13476760288256 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 166801920 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚272397608712192 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 322893357 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 $ _timestamp‚1.788423562967047e+09 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 541892608 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚366 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚13711115883008 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 541892608 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 188893153 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 127309304 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 142861824 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚299 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚301150667287552 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 171710976 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 134000202 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20201 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚78399691 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚285835077715682 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 167821400 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚301150857001472 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚287674096713216 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31168 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚13710973021184 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚13476588577280 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 171710976 trainer/global_step‚7253 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6252512256 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1243 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 189713920 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 134000204 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1261 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚48909613 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 189713920 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚262 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 188893153 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚13476760288256 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 127309672 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 6893338624 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚48909979 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚561 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 134000204 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 134000202 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6522528073 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _step‚595 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31168 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚13476760288256 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 166801920 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚272397608712192 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 322893357 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 $ _timestamp‚1.788423562967047e+09 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 541892608 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚366 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚13711115883008 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 541892608 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 188893153 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 127309304 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 142861824 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚299 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚301150667287552 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 171710976 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 134000202 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20201 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚78399691 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚285835077715682 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 167821400 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚301150857001472 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚287674096713216 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31168 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚13710973021184 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚13476588577280 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 171710976 trainer/global_step‚7253 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6252512256 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1243 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 189713920 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 134000204 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1261 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚48909613 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 189713920 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚262 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 188893153 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚13476760288256 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 127309672 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 6893338624 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚48909979 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚561 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 134000204 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 134000202 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6522528073 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31397 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 537294673 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 6893338624 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚301177200872448 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚78408655 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚322418643775488 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6343884800 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚13711508678144 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚561 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 322896366 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚13476981372416 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚13438028001786 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚561 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 549453824 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 134015140 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚301177390586368 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚322418476973568 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚23 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚287700219500032 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 188881228 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚308707135097344 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 127323566 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6252512256 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20203 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 171710976 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31168 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 322896366 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚262 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 134015140 \ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 541892608 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20203 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚7253 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚262 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31397 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚13476981372416 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚285860406702586 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 322916569 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚13477153083392 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1243 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 166801920 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 6893338624 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚13477153083392 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 188901429 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚561 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 184860760 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3428 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚48914911 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6522528073 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 134015138 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚13711365816320 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31397 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 537294673 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 6893338624 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚301177200872448 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚78408655 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚322418643775488 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6343884800 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚13711508678144 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚561 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 322896366 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚13476981372416 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚13438028001786 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚561 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 549453824 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 134015140 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚301177390586368 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚322418476973568 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚23 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚287700219500032 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 188881228 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚308707135097344 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 127323566 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6252512256 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20203 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 171710976 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31168 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 322896366 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚262ÔSº†žœ‚(:„ ŒÛäÔ–Žx gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚122.149' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•ËGWj‚(:ß ŒÛäÔ½…Àz disk.rbd1.in‚0 disk.rbd2.in‚0 network.sent‚ 1.764607e+07 network.recv‚ 2.43011e+06) disk./.usagePercent‚73.74663856987392 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.sda2.out‚ 0.0390625! disk.rbd2.out‚215.10107421875 disk.rbd0.in‚0 disk.rbd1.out‚ 3.43603515625 proc.cpu.threads‚17% disk./.usageGB‚1098.8782081604004 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0! disk.dm-0.out‚31297.130859375 disk.dm-1.out‚ 357.6640625( proc.memory.availableMB‚ 337.08203125 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.dm-0.in‚5.3125 disk.sda2.in‚0) proc.memory.percent‚17.46692657470703 cpu‚25.298249554052433 disk.sda3.out‚ 357.6640625$ disk.nvme0n1.out‚31297.130859375 proc.memory.rssMB‚ 2861.78125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125& disk.nvme0c0n1.out‚31297.130859375 disk.sda.in‚ 7.36328125 disk.sda.out‚ 357.703125 disk.dm-1.in‚7.3125$ memory_percent‚97.94261455535889 disk.nvme0n1.in‚5.3125À$g 2ž- _wandbruntime‚7048  _runtime‚7048$Pþ Ÿá¤ & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11302682757377625 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1091301292181015 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 particle/accuracy_4_of_6‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4107142984867096 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933352470397949 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4883444607257843 ( Purity/1FRt4SRqqt0FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22002895176410675 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02856740914285183 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5100694298744202 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5254262089729309 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.41734886169433594 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4469696879386902 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 jet/accuracy_1_of_6‚1 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912897944450378 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0722680389881134 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_1_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.46847501397132874 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.021768085658550262 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1851186901330948 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_4‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.39666667580604553 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.988304078578949 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.025143679231405258 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.462748259305954 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01212937943637371 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24341022968292236 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45992690324783325 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978458285331726 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.016950862482190132 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14465343952178955 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05178571119904518 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.41197311878204346 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.48960351943969727 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1370520442724228 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.29884669184684753 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09891954809427261 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5139739513397217 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16087132692337036 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0066908192820847034 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18300983309745789 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23756703734397888 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13541671633720398 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13528059422969818 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666805744171 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998248815536499 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.138314887881279 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1694769710302353 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.454113632440567 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5314525365829468 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02719910256564617 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9899314045906067 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt2SRq~–O$]uqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.43884700536727905 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19646799564361572 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_1_of_3‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9867847561836243 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 particle/accuracy_4_of_5‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.28425928950309753 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 )loss/val_total_loss‚42.92408592658989 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20574712753295898 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.35603034496307373 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.10255750268697739 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.517535924911499 particle/accuracy_1_of_5‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13167637586593628 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9493855237960815 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11188957095146179 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5008242130279541 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_6_of_7‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9912689924240112 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4217485785484314 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27524906396865845 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 )particle/specificity‚0.95328289270401 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1111111044883728 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9208037853240967 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 *jet/accuracy_1_of_3‚0.8585602045059204 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 *jet/accuracy_3_of_6‚0.1538461595773697 jet/accuracy_5_of_7‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5212215185165405 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9864781498908997 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.41143783926963806 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4493388533592224 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887130260467529 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8496196269989014 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14074888825416565 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02400478720664978 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 7validation_average_jet_accuracy‚0.42824679613113403 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.501634418964386 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4253285229206085 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964566230773926 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0671948492527008 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16087132692337036 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.21742533147335052 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1031746044754982 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1310344785451889 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 trainer/global_step‚7253 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011921104043722153 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 particle/accuracy_3_of_7‚1 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18998688459396362 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2666667103767395 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2777777910232544 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5207834243774414 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.36666667461395264 _runtime‚ 7048.75799256 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1964285671710968 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9885599613189697 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4840061068534851 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3500663638114929 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.48895660042762756 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.000933706818614155 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14249655604362488 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5337614417076111 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.977355420589447 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12441106885671616 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 jet/accuracy_4_of_6‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ( Purity/1FRt4SRqqt*FBt/FRt_purity‚0.5 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4831349551677704 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4938772916793823 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3888889253139496 particle/accuracy_5_of_7‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5102022290229797 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.13783784210681915 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17296309769153595 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.36419758200645447 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5110625624656677 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1213991791009903 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5370370745658875 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1358153223991394 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4417208731174469 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 _step‚597 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15748338401317596 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9770114421844482 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 epoch‚5 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11435960233211517 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40406978130340576 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18457092344760895 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12608566880226135 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9868363738059998 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1684361845254898 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17234452068805695 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.43591955304145813 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3561288118362427 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 )jet/accuracy_2_of_4‚0.312488317489624 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5161147117614746 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.991936445236206 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2170557677745819 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.596491277217865 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.47896137833595276 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3958333134651184 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4619675874710083 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5057535767555237 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2669565975666046 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5135706663131714 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2395431101322174 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0023969856556504965 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5269174575805664 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_3_of_4‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07965246587991714 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17129631340503693 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884235779946773e+09 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006878307089209557 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4879942834377289 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08213881403207779 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36733630299568176 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4136165976524353 jet/accuracy_3_of_7‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4268418550491333 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16572514176368713 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 particle/accuracy_7_of_7‚0.5 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.478584349155426 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8205881118774414 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7450656294822693 *jet/accuracy_2_of_3‚0.2125800997018814 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.32777777314186096 +jet/accuracy_3_of_4‚0.04091394692659378 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5597360134124756 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.29895955324172974 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4933333694934845 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_7‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 particle/accuracy_4_of_7‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10070009529590607 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05750393122434616 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9954648613929749 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4122951924800873 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.45600607991218567 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.515804648399353 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.246860072016716 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41572126746177673 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14258071780204773 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4676099717617035 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4922618567943573 +particle/sensitivity‚0.2894088923931122 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.49214980006217957 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.20454543828964233 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.024447167292237282 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 (particle/f_score‚0.37531399726867676 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9922763109207153 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2777777910232544 particle/accuracy_6_of_6‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10271720588207245 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4083179235458374 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9855191111564636 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_4‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2589285671710968 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +jet/accuracy_1_of_1‚0.48493334650993347 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5046210885047913 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.13128654658794403 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3541666567325592 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6421821713447571 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9907435774803162 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_2_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037038803100586 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.483436644077301 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9926453232765198 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24176102876663208 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.19977204501628876 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_4‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5113973617553711 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014627871103584766 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5228060483932495 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9015949964523315 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_5_of_6‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.07468637079000473 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4200650453567505 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19240227341651917 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.511947512626648 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5136063098907471 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08156028389930725 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5015432238578796 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11782940477132797 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818182468414307 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.47468581795692444 particle/accuracy_2_of_2‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.016730669885873795 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007709750905632973 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5095617175102234 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9989040493965149 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.34836676716804504 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3960966467857361 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3700792193412781 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11340747773647308 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14065656065940857 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.004032257944345474 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.29860836267471313 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25938835740089417 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5076199769973755 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.45235687494277954 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6665317416191101 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5095617175102234 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_6‚0.38461539149284363¦†«í•  æ¤ ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18300983309745789 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23756703734397888 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13541671633720398 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13528059422969818 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666805744171 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998248815536499 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.138314887881279 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1694769710302353 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.454113632440567 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5314525365829468 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02719910256564617 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9899314045906067 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.43884700536727905 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19646799564361572 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_1_of_3‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBtÌÏÎsù_purity‚0.9867847561836243 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 particle/accuracy_4_of_5‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.28425928950309753 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 )loss/val_total_loss‚42.92408592658989 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20574712753295898 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.35603034496307373 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.10255750268697739 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.517535924911499 particle/accuracy_1_of_5‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13167637586593628 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9493855237960815 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11188957095146179 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5008242130279541 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_6_of_7‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9912689924240112 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4217485785484314 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27524906396865845 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 )particle/specificity‚0.95328289270401 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1111111044883728 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9208037853240967 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 *jet/accuracy_1_of_3‚0.8585602045059204 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 *jet/accuracy_3_of_6‚0.1538461595773697 jet/accuracy_5_of_7‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5212215185165405 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9864781498908997 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.41143783926963806 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4493388533592224 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887130260467529 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8496196269989014 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14074888825416565 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02400478720664978 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 7validation_average_jet_accuracy‚0.42824679613113403 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.501634418964386 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4253285229206085 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964566230773926 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0671948492527008 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16087132692337036 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.21742533147335052 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1031746044754982 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1310344785451889 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 trainer/global_step‚7253 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011921104043722153 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 particle/accuracy_3_of_7‚1 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18998688459396362 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2666667103767395 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2777777910232544 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5207834243774414 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.36666667461395264 _runtime‚ 7048.75799256 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1964285671710968 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9885599613189697 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4840061068534851 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3500663638114929 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.48895660042762756 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.000933706818614155 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14249655604362488 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5337614417076111 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.977355420589447 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12441106885671616 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 jet/accuracy_4_of_6‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ( Purity/1FRt4SRqqt*FBt/FRt_purity‚0.5 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4831349551677704 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4938772916793823 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3888889253139496 particle/accuracy_5_of_7‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5102022290229797 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.13783784210681915 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17296309769153595 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.36419758200645447 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5110625624656677 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1213991791009903 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5370370745658875 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1358153223991394 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4417208731174469 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 _step‚597 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15748338401317596 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9770114421844482 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 epoch‚5 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11435960233211517 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40406978130340576 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18457092344760895 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12608566880226135 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9868363738059998 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1684361845254898 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17234452068805695 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.43591955304145813 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3561288118362427 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 )jet/accuracy_2_of_4‚0.312488317489624 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5161147117614746 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.991936445236206 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2170557677745819 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.596491277217865 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.47896137833595276 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3958333134651184 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4619675874710083 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5057535767555237 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2669565975666046 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5135706663131714 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2395431101322174 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0023969856556504965 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5269174575805664 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_3_of_4‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07965246587991714 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17129631340503693 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884235779946773e+09 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006878307089209557 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4879942834377289 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08213881403207779 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36733630299568176 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4136165976524353 jet/accuracy_3_of_7‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4268418550491333 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16572514176368713 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 particle/accuracy_7_of_7‚0.5 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.478584349155426 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8205881118774414 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7450656294822693 *jet/accuracy_2_of_3‚0.2125800997018814 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.32777777314186096 +jet/accuracy_3_of_4‚0.04091394692659378 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5597360134124756 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.29895955324172974 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4933333694934845 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_7‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 particle/accuracy_4_of_7‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10070009529590607 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05750393122434616 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9954648613929749 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4122951924800873 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.45600607991218567 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.515804648399353 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.246860072016716 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41572126746177673 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14258071780204773 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4676099717617035 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4922618567943573 +particle/sensitivity‚0.2894088923931122 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.49214980006217957 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.20454543828964233 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.024447167292237282 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 (particle/f_score‚0.37531399726867676 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9922763109207153 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2777777910232544 particle/accuracy_6_of_6‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10271720588207245 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4083179235458374 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9855191111564636 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_4‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2589285671710968 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +jet/accuracy_1_of_1‚0.48493334650993347 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5046210885047913 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.13128654658794403 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3541666567325592 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6421821713447571 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9907435774803162 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_2_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037038803100586 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.483436644077301 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9926453232765198 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24176102876663208 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.19977204501628876 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_4‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5113973617553711 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014627871103584766 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5228060483932495 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9015949964523315 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_5_of_6‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.07468637079000473 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4200650453567505 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19240227341651917 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.511947512626648 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5136063098907471 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08156028389930725 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5015432238578796 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11782940477132797 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818182468414307 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.47468581795692444 particle/accuracy_2_of_2‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.016730669885873795 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007709750905632973 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5095617175102234 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9989040493965149 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.34836676716804504 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3960966467857361 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3700792193412781 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11340747773647308 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14065656065940857 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.004032257944345474 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.29860836267471313 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25938835740089417 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5076199769973755 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.45235687494277954 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6665317416191101 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5095617175102234 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_6‚0.38461539149284363 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11302682757377625 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1091301292181015 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 particle/accuracy_4_of_6‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4107142984867096 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933352470397949 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4883444607257843 ( Purity/1FRt4SRqqt0FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22002895176410675 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02856740914285183 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5100694298744202 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5254262089729309 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.41734886169433594 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4469696879386902 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 jet/accuracy_1_of_6‚1 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912897944450378 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0722680389881134 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_1_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.46847501397132874 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.021768085658550262 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1851186901330948 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_4‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.39666667580604553 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.988304078578949 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.025143679231405258 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.462748259305954 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01212937943637371 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24341022968292236 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚Ù鶠ß0.45992690324783325 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978458285331726 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.016950862482190132 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14465343952178955 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05178571119904518 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.41197311878204346 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.48960351943969727 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1370520442724228 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.29884669184684753 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09891954809427261 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5139739513397217 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16087132692337036 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0066908192820847034Õ•GAŽ¡‚(:ƒ ›ÛäÔÛ–¢x gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.117' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6234 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?¸$³i¢‚(:Þ ›ÛäÔŸôè{ disk.rbd3.out‚0 disk.sda.out‚ 358.1796875 disk.nvme0n1.in‚5.3125) disk./.usagePercent‚73.74666058652186 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 cpu‚25.306298363952905 disk.dm-0.in‚5.3125# disk.dm-0.out‚31297.88818359375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125' proc.memory.availableMB‚ 219.4609375 network.sent‚ 1.7754226e+07 disk.rbd2.in‚0 disk.sda3.in‚7.3125 disk.dm-1.out‚ 358.140625 network.recv‚ 2.436287e+06 disk.rbd4.in‚0( disk.nvme0c0n1.out‚31297.88818359375 disk.sda3.out‚ 358.140625# proc.memory.rssMB‚ 2979.47265625 proc.cpu.threads‚17% disk./.usageGB‚1098.8785362243652 disk.rbd1.in‚0 disk.rbd1.out‚ 3.43603515625 disk.dm-1.in‚7.3125# memory_percent‚98.6605167388916) proc.memory.percent‚18.18525791168213& disk.nvme0n1.out‚31297.88818359375 disk.rbd2.out‚215.25 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.050781254ãQÿ›£‚(: ªÛäÔ¸þ¹x gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•±Kj¤‚(:ß ªÛäԷȲ{ disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 network.sent‚ 1.7877741e+07 disk.sda.out‚ 358.70703125 disk.sda3.in‚7.3125 cpu‚25.30575933359506) disk./.usagePercent‚73.74670513183283% disk./.usageGB‚1098.8791999816895 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125# disk.nvme0n1.out‚31563.60546875 disk.dm-1.out‚ 358.66796875 disk.rbd2.in‚0 network.recv‚ 2.442157e+06 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 3.43603515625 disk.rbd0.out‚0$ memory_percent‚96.92699909210205 disk.nvme0n1.in‚5.3125 disk.drbd1777.out‚0% disk.nvme0c0n1.out‚31563.60546875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 358.66796875 disk.rbd0.in‚0) proc.memory.percent‚17.40286350250244 disk.drbd1777.in‚ 1.03515625 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.rbd2.out‚ 215.265625# proc.memory.rssMB‚ 2851.28515625 disk.dm-0.in‚5.3125 disk.rbd4.out‚0 disk.dm-0.out‚31563.60546875( proc.memory.availableMB‚ 503.48046875B(f¥‚(:„ ¹ÛäÔÙÞx gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08Ç Ûo¦‚(:ä ¹ÛäÔÅ´{ network.recv‚ 2.446986e+06$ disk./.usageGB‚1098.878517150879( proc.memory.availableMB‚ 503.58984375 proc.cpu.threads‚17) disk./.usagePercent‚73.74665930648419 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 359.1484375 disk.sda3.in‚7.3125" proc.memory.rssMB‚ 2851.3203125 disk.rbd3.out‚0* proc.memory.percent‚17.403078079223633 disk.rbd0.in‚0$ memory_percent‚96.92633152008057 network.sent‚ 1.7884023e+07& disk.nvme0n1.out‚31829.55419921875 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda3.out‚ 359.1484375 cpu‚25.305219422717816# disk.dm-0.out‚31829.55419921875 disk.rbd2.out‚ 216.90234375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚31829.55419921875 disk.sda1.in‚ 0.05078125 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.sda.out‚359.1875 disk.rbd1.out‚ 3.43603515625嶶ï2§- _wandbruntime‚7093  _runtime‚70935·È¨Â ' lr-AdamW/pg1‚0.00032637188206148646 ' lr-AdamW/pg2‚0.00032637188206148646 trainer/global_step‚7299 $ _timestamp‚1.788423622942642e+09 _runtime‚7093.692572498 _step‚598Í š Í©Ç trainer/global_step‚7299 $ _timestamp‚1.788423622942642e+09 _runtime‚7093.692572498 _step‚598 ' lr-AdamW/pg1‚0.00032637188206148646 ' lr-AdamW/pg2‚0.00032637188206148646ÖgBú‘2ª- _wandbruntime‚7093  _runtime‚7093Ûâ„Mí'«çO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190165169 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13800880830976 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13564745705472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128266983 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13564745705472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49275155 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135006663 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13800603620864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303330249161728 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190165169 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128266317 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚274377760605184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289765275252736 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚274377777644544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78991164 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135006661 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚287903364975320 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13525363681628 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303330249161728 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289765257249792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13564973908992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49275819 _step‚599 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190139499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289765257249792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13564973908992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 trainer/global_step‚7299 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13525587330776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303330002955264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289765275252736 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325146160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135006661 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78991162 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚303330002955264 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 trainer/global_step‚7299 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13525587330776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303330002955264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289765275252736 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325146160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135006661 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78991162 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚303330002955264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135006663 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13800603620864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303330249161728 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190165169 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128266317 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚274377760605184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289765275252736 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚274377777644544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78991164 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0׬°°ô2­- _wandbruntime‚7094  _runtime‚7094RY)l´®ñM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13802747553792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚303370424813568 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190190746 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13527196081228 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135024380 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128282928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303370178593792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13566811616768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13566583399936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49282068 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13802459871232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 _step‚600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚289803613196800 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303370424813568 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128283587 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79001519 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324760083573248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚657 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190165050 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190165050 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚289803613196800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325189430 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 trainer/global_step‚7299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289803595193-@±}C856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚310957336019456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13566583399936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287682560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135024380 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13566811616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚274414063063040 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13527419730584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325215128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79001517 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚287941499832984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135024382 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚289803595193856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324759771950592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚274414080102400 % _timestamp‚1.7884236238088384e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49281411 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190190746 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303370424813568 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128283587 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79001519 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324760083573248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚657 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190165050 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190165050 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚289803613196800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325189430 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 trainer/global_step‚7299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289803595193856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚310957336019456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13566583399936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287682560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135024380 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13566811616768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚274414063063040 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13527419730584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325215128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79001517 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚287941499832984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135024382 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚289803595193856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324759771950592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚274414080102400 % _timestamp‚1.7884236238088384e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49281411 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190190746 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13527196081228 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135024380 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128282928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303370178593792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13566811616768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13566583399936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49282068 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13802459871232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 _step‚600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚289803613196800 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚305338799295488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13656644842496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚305339045501952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚632 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚291682154452992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13894402392064 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚289810121439328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚276192883497984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884236692834651e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13616997252836 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13656873046016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 327313012 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13617220901984 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326868760766464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚312974627195904 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191420401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191446071 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49599105 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚305338799295488 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135892611 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚305339045501952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚291682172455936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚276192900537344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79508914 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49599737 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135892613 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135892611 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 trainer/global_step‚7349 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129108651 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allÖ›O™ocated‚ 191446071 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191420401 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _step‚603 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚289809880750820 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135892613 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13656644842496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚7140.033581252 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13656873046016Q7zìò'ÀìO _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13617220901984 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326868760766464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚312974627195904 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191420401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191446071 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49599105 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚305338799295488 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135892611 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚305339045501952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚291682172455936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚276192900537344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79508914 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49599737 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135892613 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135892611 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 trainer/global_step‚7349 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129108651 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191446071 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191420401 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _step‚603 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚289809880750820 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135892613 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13656644842496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚7140.033581252 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13656873046016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79508912 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚305338799295488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13656644842496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚305339045501952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚632 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚291682154452992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13894402392064 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚289810121439328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚276192883497984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884236692834651e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13616997252836 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13656873046016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 327313012 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360ÛÓ¢2Á- _wandbruntime‚7140  _runtime‚7140¶Cc÷&ÂñM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚289848256296992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327381980 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚291720492397056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326912095831040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135910332 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚305379221153792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13658482536960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚7140.905329257 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191445952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291720510400000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚291720492397056 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129125811 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13658482536960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚276229202995200 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13618829652436 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129125172 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79519269 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚289848015608276 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326911794694144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13619053301792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191445952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135910330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79519267 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49605905 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚305378974934016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 327356282 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884236701546695e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49606542 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 trainer/global_step‚7349 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327381980 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79519269 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚289848015608276 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326911794694144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13619053301792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191445952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135910330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79519267 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49605905 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚305378974934016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 327356282 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884236701546695e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49606542 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 trainer/global_step‚7349 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327381980 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚289848256296992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327381980 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚291720492397056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326912095831040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135910332 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚305379221153792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13658482536960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚7140.905329257 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191445952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291720510400000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚291720492397056 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129125811 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13658482536960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚276229202995200 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13618829652436 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129125172 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ü„æx}2Ä- _wandbruntime‚7140  _runtime‚7140¼Ô*.íÅÜ trainer/global_step‚7349 2loss/SRqqt4/assignment_loss‚1.7939030198461337 /loss/FRt2/detection_loss‚1.7317154397815466 1loss/SRqqt1/detection_loss‚2.3185942359268665 1loss/SRqqt3/detection_loss‚1.9251553035796958 epoch‚6 % _timestamp‚1.7884236701565802e+09 _runtime‚7140.906383083 _step‚605 0loss/FBt2/assignment_loss‚0.1114692769650874 1loss/SRqqt2/detection_loss‚0.8859061747241546 1loss/SRqqt4/detection_loss‚2.1591086905935537 /lof„þÐuss/FBt1/detection_loss‚10.370619714260101 /loss/FBt4/detection_loss‚2.8660911321640015 2loss/FBt1/assignment_loss‚0.004281995818018913 .loss/FBt2/detection_loss‚0.750372822683018 2loss/SRqqt1/assignment_loss‚1.6041689942358062 2loss/SRqqt3/assignment_loss‚1.6505512804427045 0loss/FRt2/assignment_loss‚1.1762558138370514 0loss/FRt3/assignment_loss‚2.8913094953254417 2loss/SRqqt2/assignment_loss‚0.8671362161429201 3loss/FBt4/assignment_loss‚0.0047830021067056805 -loss/FRt3/detection_loss‚2.26248237490654 0loss/FRt1/assignment_loss‚3.1028519310057163 0loss/FRt4/assignment_loss‚3.2833079150382507 1loss/FBt3/assignment_loss‚0.03635153042462965 /loss/FRt1/detection_loss‚2.8052749873604625 /loss/FRt4/detection_loss‚2.2738792610958334 .loss/FBt3/detection_loss‚2.502083500226339 &loss/total_loss‚49.377654108490574¨ákgÆá 0loss/FRt2/assignment_loss‚1.1762558138370514 0loss/FRt3/assignment_loss‚2.8913094953254417 2loss/SRqqt2/assignment_loss‚0.8671362161429201 3loss/FBt4/assignment_loss‚0.0047830021067056805 -loss/FRt3/detection_loss‚2.26248237490654 0loss/FRt1/assignment_loss‚3.1028519310057163 0loss/FRt4/assignment_loss‚3.2833079150382507 1loss/FBt3/assignment_loss‚0.03635153042462965 /loss/FRt1/detection_loss‚2.8052749873604625 /loss/FRt4/detection_loss‚2.2738792610958334 .loss/FBt3/detection_loss‚2.502083500226339 &loss/total_loss‚49.377654108490574 trainer/global_step‚7349 2loss/SRqqt4/assignment_loss‚1.7939030198461337 /loss/FRt2/detection_loss‚1.7317154397815466 1loss/SRqqt1/detection_loss‚2.3185942359268665 1loss/SRqqt3/detection_loss‚1.9251553035796958 epoch‚6 % _timestamp‚1.7884236701565802e+09 _runtime‚7140.906383083 _step‚605 0loss/FBt2/assignment_loss‚0.1114692769650874 1loss/SRqqt2/detection_loss‚0.8859061747241546 1loss/SRqqt4/detection_loss‚2.1591086905935537 /loss/FBt1/detection_loss‚10.370619714260101 /loss/FBt4/detection_loss‚2.8660911321640015 2loss/FBt1/assignment_loss‚0.004281995818018913 .loss/FBt2/detection_loss‚0.750372822683018 2loss/SRqqt1/assignment_loss‚1.6041689942358062 2loss/SRqqt3/assignment_loss‚1.6505512804427045ÝvRÇ‚(:… „ÜäÔžýšx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.811' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5622 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q©xÈ‚(:í „ÜäÔ©­Ã{ disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625( disk.nvme0c0n1.out‚32227.52197265625 disk.rbd3.in‚0 disk.nvme0n1.in‚5.3125$ memory_percent‚96.93098068237305 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 361.69921875 disk.rbd4.in‚0* proc.memory.percent‚17.403101921081543 disk.rbd4.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.58447265625 disk.sda.out‚ 361.73828125 disk.sda3.in‚7.3125 network.sent‚ 1.8015302e+07) disk./.usagePercent‚73.74663856987392 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda.in‚ 7.36328125& disk.nvme0n1.out‚32227.52197265625# disk.dm-0.out‚32227.52197265625& proc.memory.availableMB‚ 502.828125# proc.memory.rssMB‚ 2851.32421875% disk./.usageGB‚1098.8782081604004 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚5.3125 cpu‚25.302588011669076 proc.cpu.threads‚17 disk.sda3.out‚ 361.69921875 network.recv‚ 2.468461e+06! disk.rbd2.out‚217.22705078125 disk.rbd0.in‚01½=cŽÉ‚(:ƒ “ÜäÔöËx gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚104.16' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.832 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠÕP rÊ‚(:ç “ÜäÔ¿¹³{ disk.sda1.out‚0 disk.sda2.in‚0 cpu‚25.302027664379406 disk.sda.out‚ 362.34375 disk.rbd1.in‚0 disk.dm-0.in‚5.3125# disk.dm-0.out‚32227.87158203125 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd1.out‚ 3.58447265625 disk.dm-1.out‚ 362.3046875! disk.rbd2.out‚218.80126953125 network.recv‚ 2.475388e+06 disk.rbd4.in‚0 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚32227.87158203125& proc.memory.availableMB‚ 502.453125# proc.memory.rssMB‚ 2851.32421875 disk.rbd3.in‚0 network.sent‚ 1.812386e+07% disk./.usageGB‚1098.8782081604004 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.sda3.out‚ 362.3046875 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚32227.87158203125 disk.sda1.in‚ 0.05078125$ memory_percent‚96.93326950073242) disk./.usagePercent‚73.74663856987392 disk.rbd3.out‚0 disk.rbd0.in‚0* proc.memory.percent‚17.403101921081543 proc.cpu.threads‚17KžÂµŽË‚(:ƒ ¢ÜäÔþºy gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.405' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.881 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%,‡rÌ‚(:ç ¢ÜäÔ“ƒËz) disk./.usagePercent‚73.74664215397941 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 362.90625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0# proc.memory.rssMB‚ 2851.32421875 cpu‚25.301509350945732& disk.nvme0n1.out‚32327.03759765625# disk.dm-0.out‚32327.03759765625 proc.cpu.threads‚17 disk.rbd2.in‚0! disk.rbd2.out‚218.81689453125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚32327.03759765625 disk.rbd3.in‚0 disk.sda.out‚ 362.9453125 disk.drbd1777.out‚0 network.recv‚ 2.486668e+06 disk.rbd1.out‚ 3.61669921875 disk.dm-1.in‚7.3125 network.sent‚ 1.8133608e+07 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0$ disk./.usageGB‚1098.878261566162 disk.sda.in‚ 7.36328125 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 362.90625$ memory_percent‚96.93214893341064( proc.memory.availableMB‚ 502.63671875* proc.memory.percent‚17.403101921081543 disk.sda3.in‚7.3125¨Þ9¥2Í- _wandbruntime‚7186  _runtime‚7186ééaÅο & lr-AdamW/pg1‚0.0003150253041455998 & lr-AdamW/pg2‚0.0003150253041455998 trainer/global_step‚7399 # _timestamp‚1.78842371565666e+09 _runtime‚7186.406603797 _step‚606ñÅ’¾ÊÏÄ trainer/global_step‚7399 # _timestamp‚1.78842371565666e+09 _runtime‚7186.406603797 _step‚606 & lr-AdamW/pg1‚0.0003150253041455998 & lr-AdamW/pg2‚0.0003150253041455998ÞWöÊC2Ð- _wandbruntime‚7186  _runtime‚7186¼Ÿ£»í'ÑçO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307347595635712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _runtime‚7186.407653706 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136778561 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192726973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚315033135948800 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13987565391360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129946488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚329020978550272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚315033112008704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278008023430144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13748772183040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚293599069659136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13987842601472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192701303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚7399 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80026664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚329020677400064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚293599051656192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49920490 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136778563 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329505536 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13748543979520 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136778561 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚291716877903336 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136778563 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192726973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13748772183040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192701303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0ínò'ÒìO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136778563 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192726973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13748772183040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192701303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307347595635712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _runtime‚7186.407653706 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136778561 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192726973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚315033135948800 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13987565391360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129946488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚329020978550272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚315033112008704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278008023430144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13748772183040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚293599069659136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13987842601472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192701303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚7399 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80026664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚329020677400064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚293599051656192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49920490 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136778563 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329505536 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13748543979520 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136778561 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚291716877903336 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299ßóŠ32Ó- _wandbruntime‚7187  _runtime‚7187\`¥¤¨ÔñM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚307388017494016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚291755012761000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49926347 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13750609890816 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13750381673984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚307388017494016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136796280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293637389600256 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136796280 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚293637407603200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278044325888000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136796282 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129963364 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _step‚608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocab?:Oted‚13710686873000 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136796282 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13989705286656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13710463223644 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚291754772072284 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329548832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚329063716086272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192752550 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329523134 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329548832 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329523134 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278044308848640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚293637407603200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278044325888000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136796282 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129963364 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _step‚608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13710686873000 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136796282 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13989705286656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13710463223644 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚291754772072284 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329548832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚329063716086272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192752550 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329523134 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329548832 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329523134 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278044308848640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13750381673984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚307388017494016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262ào˜Ä2Ö- _wandbruntime‚7187  _runtime‚7187‰úböU×Ï % _timestamp‚1.7884237165364728e+09 /loss/FRt1/assignment_loss‚3.579848463942365 /loss/FRt4/assignment_loss‚3.526726606346312 2loss/SRqqt4/assignment_loss‚1.7813836139963661 /loss/FRt1/detection_loss‚2.3502698329890648 _step‚609 1loss/FBt4/assignment_loss‚0.21774688833910558 /loss/FRt3/detection_loss‚2.8159117733730987 -loss/FBt2/detection_loss‚0.58425412499169 trainer/global_step‚7399 0loss/FBt3/assignment_loss‚0.6776115395128727 /loss/FBt1/detection_loss‚2.9812606147357394 1loss/SRqqt3/detection_loss‚1.9922623869445588 %loss/total_loss‚44.31907240384136 _runtime‚ 7187.28630643 .loss/FRt3/assignment_loss‚3.51277652908774 1loss/SRqqt1/assignment_loss‚1.744105105055496 2loss/SRqqt3/assignment_loss‚1.3318672308491335 1loss/FBt2/assignment_loss‚0.21718218450169088 .loss/FRt2/detection_loss‚1.806340639013797 .loss/FBt3/detection_loss‚2.513323724269867 .loss/FBt4/detection_loss‚2.021612763404846 2loss/SRqqt2/assignment_loss‚0.9438199311873671 1loss/FBt1/assignment_loss‚0.13382076259170259 .loss/FRt4/detection_loss‚2.449117135434017 0loss/SRqqt1/detection_loss‚2.537946520373225 1loss/SRqqt2/detection_loss‚1.0196194848898104 0loss/FRt2/assignment_loss‚1.6410668802758057 1loss/SRqqt4/detection_loss‚1.9391976677356881 epoch‚6.C vZØÔ % _timestamp‚1.7884237165364728e+09 /loss/FRt1/assignment_loss‚3.579848463942365 /loss/FRt4/assignment_loss‚3.526726606346312 2loss/SRqqt4/assignment_loss‚1.7813836139963661 /loss/FRt1/detection_loss‚2.3502698329890648 _step‚609 1loss/FBt4/assignment_loss‚0.21774688833910558 /loss/FRt3/detection_loss‚2.8159117733730987 -loss/FBt2/detection_loss‚0.58425412499169 trainer/global_step‚7399 0loss/FBt3/assignment_loss‚0.6776115395128727 /loss/FBt1/detection_loss‚2.9812606147357394 1loss/SRqqt3/detection_loss‚1.9922623869445588 %loss/total_loss‚44.31907240384136 _runtime‚ 7187.28630643 .loss/FRt3/assignment_loss‚3.51277652908774 1loss/SRqqt1/assignment_loss‚1.744105105055496 2loss/SRqqt3/assignment_loss‚1.3318672308491335 1loss/FBt2/assignment_loss‚0.21718218450169088 .loss/FRt2/detection_loss‚1.806340639013797 .loss/FBt3/detection_loss‚2.513323724269867 .loss/FBt4/detection_loss‚2.021612763404846 2loss/SRqqt2/assignment_loss‚0.9438199311873671 1loss/FBt1/assignment_loss‚0.13382076259170259 .loss/FRt4/detection_loss‚2.449117135434017 0loss/SRqqt1/detection_loss‚2.537946520373225 1loss/SRqqt2/detection_loss‚1.0196194848898104 0loss/FRt2/assignment_loss‚1.6410668802758057 1loss/SRqqt4/detection_loss‚1.9391976677356881 epoch‚6áÚ¬ VÙ‚(:… ±ÜäÔ´éˆx gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.999' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á97tÚ‚(:é ±ÜäÔªŠÀz disk.rbd2.in‚0! disk.rbd2.out‚218.87939453125 disk.rbd1.in‚0 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0& disk.nvme0c0n1.out‚32426.357421875 disk.dm-0.in‚5.3125! disk.dm-0.out‚32426.357421875 network.recv‚ 2.491447e+06 disk.drbd1777.in‚ 1.03515625 disk.sda3.out‚ 363.40234375 disk.sda2.out‚ 0.0390625# proc.memory.rssMB‚ 2851.32421875 cpu‚25.300987959508266 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.rbd1.out‚ 3.61669921875 disk.dm-1.out‚ 363.40234375$ memory_percent‚96.93429470062256) disk./.usagePercent‚73.74664240998693 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.403101921081543 network.sent‚ 1.8139902e+07% disk./.usageGB‚1098.8782653808594 disk.rbd4.in‚0 disk.sda.out‚ 363.44140625 disk.dm-1.in‚7.3125( proc.memory.availableMB‚ 502.28515625 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚32426.357421875|½ÇÂÛ‚(:„ ÀÜäÔûŸ˜x gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.299 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gÁBoÜ‚(:ä ÀÜäÔ–¢ê{ disk.rbd2.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125 network.recv‚ 2.496271e+06 disk.sda.in‚ 7.36328125$ disk.nvme0n1.out‚32525.451171875 disk.rbd0.in‚0 disk.rbd4.out‚0$ memory_percent‚96.93286418914795) disk./.usagePercent‚73.74664036192667 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda3.out‚ 363.84765625 disk.rbd4.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0! disk.rbd2.out‚218.89501953125 disk.sda1.out‚0 disk.dm-0.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.sda3.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0( proc.memory.availableMB‚ 502.51953125# proc.memory.rssMB‚ 2851.32421875% disk./.usageGB‚1098.8782348632812 disk.rbd1.out‚ 3.6796875 disk.nvme0n1.in‚5.3125 network.sent‚ 1.8246594e+07 disk.dm-1.out‚ 363.84765625& disk.nvme0c0n1.out‚32525.451171875* proc.memory.percent‚17.403101921081543 cpu‚25.30042391398387 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda.out‚ 363.88671875! disk.dm-0.out‚32525.451171875gÁ±VÝ‚(:… ÏÜäÔŠžÃx gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.324' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4648 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0diYtÞ‚(:é ÏÜäÔ¥ô‘| disk.sda3.out‚ 364.21484375 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125$ memory_percent‚96.93286418914795( proc.memory.availableMB‚ 502.51953125) disk./.usagePercent‚73.74664189797187 disk.dm-0.in‚5.3125 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚ 364.25390625 cpu‚25.29993848572516% disk./.usageGB‚1098.8782577514648# disk.dm-0.out‚32525.85888671875 disk.dm-1.out‚ 364.21484375 disk.drbd1777.out‚0 disk.rbd4.in‚0( disk.nvme0c0n1.out‚32525.85888671875# proc.memory.rssMB‚ 2851.32421875 network.recv‚ 2.500567e+06 disk.dm-1.in‚7.3125 disk.rbd2.out‚219.0478515625 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 proc.cpu.threads‚17 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚32525.85888671875 disk.sda2.in‚0* proc.memory.percent‚17.403101921081543 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125 disk.sda1.out‚0 disk.rbd3.out‚0 network.sent‚ 1.8252654e+07 disk.rbd1.out‚ 3.6796875 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0e?%2ß- _wandbruntime‚7233  _runtime‚7233©Î—üÉàà ' lr-AdamW/pg2‚0.00030937835021345016 trainer/global_step‚7449 % _timestamp‚1.7884237622926686e+09 _runtime‚7233.042642034 _step‚610 ' lr-AdamW/pg1‚0.00030937835021345016JCIÞÎáÈ ' lr-AdamW/pg1‚0.00030937835021345016 ' lr-AdamW/pg2‚0.00030937835021345016 trainer/global_step‚7449 % _timestamp‚1.7884237622926686e+09 _runtime‚7233.042642034 _step‚610â!©|)2â- _wandbruntime‚7233  _runtime‚7233•±pc%ãçO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚645 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193982205 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 331646716 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130789412 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331172973218304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚309356638182400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317091620761600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137664513 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚279823146322944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194007875 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50245643 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚279823129283584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13800264395252 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309356638182400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚293623393678836 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 trainer/global_step‚7449 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309356391975936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331172661582336 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 194007875 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137664513 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331672388 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13840671320064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _step‚611 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193982205 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317091596821504 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80544412 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130790057 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚643 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 311635968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13800488044400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 _runtime‚7233.043662076 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13840443116544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚293623634367344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.curEÞµŠrent‚262 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚295515966862336 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13840671320064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50245000 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80544412 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130790057 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚643 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 311635968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13800488044400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 _runtime‚7233.043662076 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13840443116544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚293623634367344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚295515966862336 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13840671320064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50245000 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚309356638182400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317091620761600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137664513 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚279823146322944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194007875 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50245643 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚279823129283584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13800264395252 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309356638182400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚293623393678836 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 trainer/global_step‚7449 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309356391975936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331172661582336 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 194007875 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137664513 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331672388 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13840671320064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _step‚611 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193982205 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317091596821504 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50250978 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331215997578752 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚7233.915278088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884237631646137e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13842509027840 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13842280811008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14082932021248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚309396813834240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚317132790457856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13802320444208 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331689986 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331715684 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130806382 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137682230 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194033452 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13842280811008 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚293661528536292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80554769 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50251613 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚7449 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137682232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295554286803456 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137682230 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130805745 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚295554304806400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331715684 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295554286803456 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137682230 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130805745 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚295554304806400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331715684 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50250978 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331215997578752 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚7233.915278088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884237631646137e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13842509027840 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13842280811008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14082932021248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚309396813834240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚317132790457856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13802320444208 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331689986 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331715684 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130806382 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137682230 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194033452 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13842280811008 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚293661528536292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80554769 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50251613 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚7449 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137682232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚311365188316160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195263107 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 286470144 % _timestamp‚1.7884238089728692e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13932570457088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195263107 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚297432864065536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 333839240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 333813568 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13932342253568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 262530048 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138550463 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 trainer/global_step‚7499 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚311365434522624 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138550461 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚297432846062592 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13932342253568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 262530048 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138550463 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 trainer/global_step‚7499 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚311365434522624 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138550461 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚297432846062592 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚311365188316160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195263107 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 286470144 % _timestamp‚1.7884238089728692e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13932570457088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195263107 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚297432864065536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 333839240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 333813568 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 333882536 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319191251330560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195314354 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13934408164864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295568285000300 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138568182 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138568180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚333368058156032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚281674554634240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 333856838 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138568182 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚297471184006656 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13934179948032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13893730366060 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7280.625251362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319191275270656 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295568525689016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 333856838 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚297471202009600 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195288658 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31—@ëÆ<397 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50574934 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131647451 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚281674571673600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13893954015416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13934179948032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚311405610174464 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14176782885376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚297471184006656 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138568180 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131648037 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 trainer/global_step‚7499 % _timestamp‚1.7884238098748808e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 333882536 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚311405610174464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81072517 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14176516174336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195314354 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131647451 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚281674571673600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13893954015416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13934179948032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚311405610174464 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14176782885376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚297471184006656 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138568180 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131648037 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 trainer/global_step‚7499 % _timestamp‚1.7884238098748808e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 333882536 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚311405610174464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81072517 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14176516174336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195314354 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 333882536 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319191251330560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195314354 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13934408164864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295568285000300 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138568182 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138568180 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚333368058156032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚281674554634240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 333856838 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138568182 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚297471184006656 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13934179948032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13893730366060 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7280.625251362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319191275270656 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295568525689016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 333856838 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚297471202009600 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992èlVœ›2ú- _wandbruntime‚7280  _runtime‚7280N TÆ[ûÕ 1loss/SRqqt4/detection_loss‚1.9832343090982998 2loss/SRqqt2/assignment_loss‚0.8284329171774203 0loss/SRqqt2/detection_loss‚1.029885433550025 /loss/FBt1/detection_loss‚1.8970793584982555 &loss/total_loss‚44.720424333978826 _runtime‚7280.626423099 .loss/FBt3/detection_loss‚2.908006658156713 epoch‚6 3loss/FBt3/assignment_loss‚0.0007783872012320595 0loss/FBt4/assignment_loss‚1.9986013760353671 -loss/FRt1/detection_loss‚2.59350204226133 .loss/FBt4/detection_loss‚2.978324234485626 /loss/FRt1/assignment_loss‚3.525747808250221 0loss/FRt2/assignment_loss‚1.5070472705364226 0loss/FRt3/assignment_loss‚2.7325130864977836 2loss/SRqqt1/assignment_loss‚1.9714813610272748 1loss/SRqqt3/assignment_loss‚1.590787234082818 /loss/FRt4/detection_loss‚2.1540828439823296 /loss/FBt2/detection_loss‚0.5826419618394639 trainer/global_step‚7499 2loss/SRqqt4/assignment_loss‚1.4752274537276404 0loss/FBt1/assignment_loss‚0.4061341865453869 0loss/FBt2/assignment_loss‚0.1441691307681316 /loss/FRt2/detection_loss‚1.6109416829049588 1loss/SRqqt1/detection_loss‚2.8113105573824475 1loss/SRqqt3/detection_loss‚1.8164336216449737 % _timestamp‚1.7884238098766077e+09 _step‚617 0loss/FRt4/assignment_loss‚3.5703637152910233 .loss/FRt3/detection_loss‚2.603697703033686ÂíIú`üÚ /loss/FRt1/assignment_loss‚3.525747808250221 0loss/FRt2/assignment_loss‚1.5070472705364226 0loss/FRt3/assignment_loss‚2.7325130864977836 2loss/SRqqt1/assignment_loss‚1.9714813610272748 1loss/SRqqt3/assignment_loss‚1.590787234082818 /loss/FRt4/detection_loss‚2.1540828439823296 /loss/FBt2/detection_loss‚0.5826419618394639 trainer/global_step‚7499 2loss/SRqqt4/assignment_loss‚1.4752274537276404 0loss/FBt1/assignment_loss‚0.4061341865453869 0loss/FBt2/assignment_loss‚0.1441691307681316 /loss/FRt2/detection_loss‚1.6109416829049588 1loss/SRqqt1/detection_loss‚2.8113105573824475 1loss/SRqqt3/detection_loss‚1.8164336216449737 % _timestamp‚1.7884238098766077e+09 _step‚617 0loss/FRt4/assignment_loss‚3.5703637152910233 .loss/FRt3/detection_loss‚2.603697703033686 1loss/SRqqt4/detection_loss‚1.9832343090982998 2loss/SRqqt2/assignment_loss‚0.8284329171774203 0loss/SRqqt2/detection_loss‚1.029885433550025 /loss/FBt1/detection_loss‚1.8970793584982555 &loss/total_loss‚44.720424333978826 _runtime‚7280.626423099 .loss/FBt3/detection_loss‚2.908006658156713 epoch‚6 3loss/FBt3/assignment_loss‚0.0007783872012320595 0loss/FBt4/assignment_loss‚1.9986013760353671 -loss/FRt1/detection_loss‚2.59350204226133 .loss/FBt4/detection_loss‚2.978324234485626éQç[‰ý‚(:„ ‹ÝäÔï¤x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.957' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶™ùzþ‚(:ï ‹ÝäÔ²Ó×{ disk.sda3.out‚ 366.29296875 disk.rbd1.out‚ 3.74267578125 disk.nvme0n1.in‚5.3125$ memory_percent‚97.95167446136475 network.recv‚ 2.526728e+06 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0& disk.nvme0n1.out‚32824.27587890625 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.8382517e+07 disk.rbd3.out‚0 disk.dm-1.out‚ 366.29296875# proc.memory.rssMB‚ 2851.32421875) disk./.usagePercent‚73.71576918139522 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125* proc.memory.percent‚17.403101921081543! disk.rbd2.out‚220.74365234375 disk.rbd4.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚7.3125 disk.rbd0.in‚0% disk./.usageGB‚1098.4182319641113 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚32824.27587890625# disk.dm-0.out‚32824.27587890625 disk.rbd0.out‚0 cpu‚25.297823748877803 disk.rbd4.in‚0 disk.dm-0.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0( proc.memory.availableMB‚ 335.59765625 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 366.33203125“$ÇŽÿ‚(:ƒ šÝäԼܡx gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑœSßq€ ‚(:æ šÝäÔŤè{ disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.dm-0.in‚5.3125# disk.dm-0.out‚32923.44775390625 disk.sda.out‚ 366.9453125& disk.nvme0n1.out‚32923.44775390625" memory_percent‚97.953200340271% disk./.usageGB‚1098.4182319641113! disk.rbd2.out‚220.75537109375 disk.rbd0.in‚0 disk.sda2.out‚ 0.0390625* proc.memory.percent‚17.403101921081543 proc.cpu.threads‚17 network.recv‚ 2.533779e+06 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.rbd1.in‚0 disk.dm-1.in‚7.3125 disk.rbd4.out‚0( disk.nvme0c0n1.out‚32923.44775390625 disk.rbd1.out‚ 3.74267578125( proc.memory.availableMB‚ 335.34765625 network.sent‚ 1.8490833e+07 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚5.3125 disk.rbd0.out‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 366.90625 disk.sda2.in‚0# proc.memory.rssMB‚ 2851.32421875) disk./.usagePercent‚73.71576918139522 disk.dm-1.out‚ 366.90625 disk.sda1.out‚0 disk.rbd3.in‚0 cpu‚25.297281649475483w†ÛL ‚(:‚ ©ÝäÔý›x gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚90.86' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„¥&aq‚ ‚(:æ ©ÝäÔ—¬üz network.recv‚ 2.538795e+06% disk./.usageGB‚1098.4182319641113 disk.rbd3.out‚0 disk.rbd1.out‚ 3.74267578125 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0( proc.memory.availableMB‚ 335.59765625# proc.memory.rssMB‚ 2851.32421875 cpu‚25.296782718675388 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd4.in‚0 disk.rbd4.out‚0) disk./.usagePercent‚73.71576918139522 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 367.2734375 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.dm-1.out‚ 367.2734375* proc.memory.percent‚17.403101921081543 disk.sda3.in‚7.3125% disk.nvme0n1.out‚33022.7236328125" disk.dm-0.out‚33022.7236328125! disk.rbd2.out‚220.81787109375 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.nvme0c0n1.in‚5.3125 network.sent‚ 1.8497425e+07 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚367.3125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚33022.7236328125$ memory_percent‚97.95167446136475u~c¸2ƒ - _wandbruntime‚7326  _runtime‚7326ËD1 Ç„ Á & lr-AdamW/pg2‚0.0002981429962391695 trainer/global_step‚7549 % _timestamp‚1.7884238555531213e+09 _runtime‚7326.303063363 _step‚618 & lr-AdamW/pg1‚0.0002981429962391695w‹¸<Ì… Æ & lr-AdamW/pg1‚0.0002981429962391695 & lr-AdamW/pg2‚0.0002981429962391695 trainer/global_step‚7549 % _timestamp‚1.7884238555531213e+09 _runtime‚7326.303063363 _step‚618êéçÈg2† - _wandbruntime‚7326  _runtime‚7326¼=Ï í'‡ çO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196544009 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139436413 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚321208566447104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚299349743265792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14024241390592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50892138 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚335476710607872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313373984656384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283453375069184 trainer/global_step‚7549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81579912 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299349761268736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14024469594112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 % _timestamp‚1.7884238555540445e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196544009 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚313374230862848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 _step‚619 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 335980420 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚313373984656384 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14024469594112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196569679 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313373984656384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283453375069184 trainer/global_step‚7549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81579912 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299349761268736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14024469594112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 % _timestamp‚1.7884238555540445e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196544009 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚313374230862848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 _step‚619 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 335980420 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚313373984656384 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14024469594112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196569679 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50892138 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚335476710607872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196595256 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196595256 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299388081209856 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚313414160294912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139454132 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _runtime‚7327.189753309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336049388 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚299388099212800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50897750 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13985587586624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81590267 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283489694566400 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚313414406514688 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335520046169088 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 % _timestamp‚1.7884238564392126e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132488622 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196569560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335519745032192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13985363937268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336023690 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 336023690 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14270008888832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚313414406514688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81590269 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139454130 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196569560 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336049388 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139454132 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚297475282153024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299388081209856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50898353 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 trainer/global_step‚7549 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14026307301888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚299388099212800 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139454130 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚321249736143360 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132488017 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14026079085056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚297475041464308 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _step‚620 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚321249760083456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚603 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262︆ü&‹ öM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81590269 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139454130 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196569560 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336049388 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139454132 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚297475282153024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299388081209856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50898353 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 trainer/global_step‚7549 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14026307301888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚299388099212800 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139454130 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚321249736143360 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132488017 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14026079085056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚297475041464308 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _step‚620 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚321249760083456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚603 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14026079085056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14026307301888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196595256 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196595256 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299388081209856 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚313414160294912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139454132 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _runtime‚7327.189753309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336049388 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚299388099212800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50897750 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13985587586624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81590267 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283489694566400 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚313414406514688 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335520046169088 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 % _timestamp‚1.7884238564392126e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132488622 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196569560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335519745032192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13985363937268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336023690 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 336023690 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14270008888832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚313414406514688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262ì@ÙŽ2Œ - _wandbruntime‚7327  _runtime‚7327ÝÝe\ Ö 0loss/FRt1/assignment_loss‚3.3401054841716116 0loss/SRqqt3/detection_loss‚1.753948487341404 _step‚621 1loss/FBt4/assignment_loss‚0.06417910640852319 epoch‚6 /loss/FRt2/assignment_loss‚1.356210323267205 0loss/FRt3/assignment_loss‚2.6581230468866304 2loss/SRqqt2/assignment_loss‚1.1083628417303164 2loss/SRqqt4/assignment_loss‚1.0989565518410767 2loss/FBt1/assignment_loss‚0.056959141788502164 .loss/FRt4/detection_loss‚2.285702365055144 /loss/FBt4/detection_loss‚2.3991208341386585 trainer/global_step‚7549 % _timestamp‚1.7884238564410138e+09 2loss/SRqqt1/assignment_loss‚1.4489704723711367 1loss/SRqqt2/detection_loss‚1.4528422449149339 /loss/FBt2/detection_loss‚0.5182267496013082 _runtime‚7327.190859583 1loss/FBt2/assignment_loss‚0.14889114432115624 .loss/FRt1/detection_loss‚2.582070720631902 .loss/FBt1/detection_loss‚4.136755251884461 .loss/FBt3/detection_loss‚2.486309136663164 /loss/FRt4/assignment_loss‚3.307228186395433 1loss/FBt3/assignment_loss‚0.08560015273230549 /loss/FRt3/detection_loss‚2.3509739332082793 0loss/SRqqt1/detection_loss‚2.620482937053398 1loss/SRqqt4/detection_loss‚1.9139070575143777 2loss/SRqqt3/assignment_loss‚1.4772017960963042 /loss/FRt2/detection_loss‚2.1901154253025386 %loss/total_loss‚42.84124339131977ÞP7aŽ Û /loss/FBt4/detection_loss‚2.3991208341386585 trainer/global_step‚7549 % _timestamp‚1.7884238564410138e+09 2loss/SRqqt1/assignment_loss‚1.4489704723711367 1loss/SRqqt2/detection_loss‚1.4528422449149339 /loss/FBt2/detection_loss‚0.5182267496013082 _runtime‚7327.190859583 1loss/FBt2/assignment_loss‚0.14889114432115624 .loss/FRt1/detection_loss‚2.582070720631902 .loss/FBt1/detection_loss‚4.136755251884461 .loss/FBt3/detection_loss‚2.486309136663164 /loss/FRt4/assignment_loss‚3.307228186395433 1loss/FBt3/assignment_loss‚0.08560015273230549 /loss/FRt3/detection_loss‚2.3509739332082793 0loss/SRqqt1/detection_loss‚2.620482937053398 1loss/SRqqt4/detection_loss‚1.9139070575143777 2loss/SRqqt3/assignment_loss‚1.4772017960963042 /loss/FRt2/detection_loss‚2.1901154253025386 %loss/total_loss‚42.84124339131977 0loss/FRt1/assignment_loss‚3.3401054841716116 0loss/SRqqt3/detection_loss‚1.753948487341404 _step‚621 1loss/FBt4/assignment_loss‚0.06417910640852319 epoch‚6 /loss/FRt2/assignment_loss‚1.356210323267205 0loss/FRt3/assignment_loss‚2.6581230468866304 2loss/SRqqt2/assignment_loss‚1.1083628417303164 2loss/SRqqt4/assignment_loss‚1.0989565518410767 2loss/FBt1/assignment_loss‚0.056959141788502164 .loss/FRt4/detection_loss‚2.285702365055144í˜G]< ‚(:‚ ¸ÝäÔäñ£x gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚98.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.646 gpu.0.smClock‚1410 gpu.0.memoryClofôÔ Qck‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03f™o ‚(:ä ¸ÝäÔ™Ž†{ disk.sda3.in‚7.3125 disk.sda2.in‚0 cpu‚25.29624534703851 network.sent‚ 1.850274e+07 disk.sda1.out‚0" disk.dm-0.out‚33023.1220703125( proc.memory.availableMB‚ 335.72265625 proc.cpu.threads‚17 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625) disk./.usagePercent‚73.71576918139522 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚7.3125 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 3.74267578125 disk.dm-1.out‚ 367.796875$ memory_percent‚97.95091152191162* proc.memory.percent‚17.403101921081543 network.recv‚ 2.54223e+06 disk.rbd4.in‚0% disk.nvme0n1.out‚33023.1220703125% disk./.usageGB‚1098.4182319641113 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd2.in‚0# proc.memory.rssMB‚ 2851.32421875 disk.dm-0.in‚5.3125 disk.drbd1777.in‚ 1.03515625' disk.nvme0c0n1.out‚33023.1220703125 disk.sda.in‚ 7.36328125! disk.rbd2.out‚222.34521484375 disk.sda.out‚ 367.8359375 disk.sda3.out‚ 367.796875 disk.rbd1.in‚0 disk.drbd1777.out‚0Ž1¶1‘ ‚(:… ÇÝäÔß›íy gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0< ór’ ‚(:ç ÇÝäÔ¡¾™|% disk./.usageGB‚1098.4183158874512 disk.sda3.out‚ 368.16796875 disk.drbd1777.out‚0$ memory_percent‚97.95060157775879# proc.memory.rssMB‚ 2851.32421875 disk.nvme0c0n1.in‚5.3125 network.sent‚ 1.8611108e+07) disk./.usagePercent‚73.71577481356097 disk.rbd1.out‚ 3.74267578125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd3.in‚0& disk.nvme0c0n1.out‚33122.611328125! disk.rbd2.out‚222.47021484375 disk.sda1.in‚ 0.05078125 proc.cpu.threads‚17 disk.sda.in‚ 7.36328125$ disk.nvme0n1.out‚33122.611328125 disk.sda2.in‚0 disk.rbd3.out‚0 cpu‚25.295736454821274 disk.sda.out‚ 368.20703125 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 368.16796875 disk.sda1.out‚0' proc.memory.availableMB‚ 335.7734375 network.recv‚ 2.55016e+06 disk.nvme0n1.in‚5.3125! disk.dm-0.out‚33122.611328125 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.403101921081543 disk.sda3.in‚7.3125 disk.dm-1.in‚7.3125 disk.rbd2.in‚0T0_R“ ‚(:… ÖÝäÔ؇‹x gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SÓ8q” ‚(:æ ÖÝäÔš²Àz! disk.dm-0.out‚33221.681640625$ memory_percent‚97.95222282409668 network.recv‚ 2.554145e+06% disk./.usageGB‚1098.4182968139648 disk.rbd2.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 3.74267578125 disk.rbd4.in‚0 disk.sda.out‚ 368.82421875$ disk.nvme0n1.out‚33221.681640625 disk.sda2.in‚0 disk.dm-1.in‚7.3125! disk.rbd2.out‚222.48583984375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125 disk.dm-1.out‚ 368.78515625 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 368.78515625 disk.nvme0n1.in‚5.3125* proc.memory.percent‚17.403101921081543 network.sent‚ 1.8616925e+07 disk.sda1.out‚0# proc.memory.rssMB‚ 2851.32421875 cpu‚25.29524978652645( disk./.usagePercent‚73.7157735335233 disk.rbd0.out‚0& disk.nvme0c0n1.out‚33221.681640625 disk.sda3.in‚7.3125 disk.dm-0.in‚5.3125 disk.sda2.out‚ 0.0390625' proc.memory.availableMB‚ 335.5078125 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd1.in‚0À: 2• - _wandbruntime‚7372  _runtime‚7372É*Ô*É– à % _timestamp‚1.7884239017790663e+09 _runtime‚7372.528985228 _step‚622 ' lr-AdamW/pg1‚0.00029255693762796683 ' lr-AdamW/pg2‚0.00029255693762796683 trainer/global_step‚7599Æp2¼Î— È ' lr-AdamW/pg1‚0.00029255693762796683 ' lr-AdamW/pg2‚0.00029255693762796683 trainer/global_step‚7599 % _timestamp‚1.7884239017790663e+09 _runtime‚7372.528985228 _step‚622îBävr2˜ - _wandbruntime‚7372  _runtime‚7372÷òí'™ çO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚301266658471936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337629052407296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚299343903759368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82097664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197824911 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14075165108876 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚301266640468992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚315383027203072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82097662 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14075388758024 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚299343663070860 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140322363 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14116368731136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 197824911 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133311841 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323267075200000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285268515001344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚315383027203072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14116140527616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14116140527616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140322363 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14361977207296 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285268497961984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚315382780996608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197850581 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140322361 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14116368731136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51214824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 trainer/global_step‚7599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 338147272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140322361 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 338172944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚301266640468992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51214179 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 _step‚623 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 197850581 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚315382780996608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚301266658471936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 _runtime‚7372.530058591 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 133312488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚337628761742848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 % _timestamp‚1.7884239017798202e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚647 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14361710482944zÝò'š ìO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 trainer/global_step‚7599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 338147272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140322361 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 338172944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚301266640468992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51214179 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 _step‚623 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 197850581 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚315382780996608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚301266658471936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 _runtime‚7372.530058591 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 133312488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚337628761742848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 % _timestamp‚1.7884239017798202e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚647 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14361710482944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚301266658471936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337629052407296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚299343903759368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82097664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197824911 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14075165108876 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚301266640468992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚315383027203072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82097662 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14075388758024 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚299343663070860 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140322363 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14116368731136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 197824911 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133311841 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323267075200000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285268515001344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚315383027203072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14116140527616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14116140527616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140322363 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14361977207296 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285268497961984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚315382780996608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197850581 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140322361 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14116368731136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51214824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561ïæ˜*2› - _wandbruntime‚7373  _runtime‚73734êqßÏœ ðM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 338190542 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14077221157832 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚323308244896256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14076997508476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337672095742464 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 trainer/global_step‚7599 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14363850846208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140340082 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚301304978413056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51220981 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 $ _timestamp‚1.788423902640116e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14363577843712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚301304978413056 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14117978222080 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133329000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚299382038617032 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚299381797928316 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚301304996416000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14117978222080 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133328374 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285304817459200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚301304996416000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197850462 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338216240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14118206438912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚323308220956160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚315423202854912 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51220357 _step‚624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 338190542 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197876158 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140340080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82108017 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140340080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚7373.390947021 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14118206438912 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197850462 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 197876158 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561¦è˜û& õM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140340080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚7373.390947021 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14118206438912 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197850462 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 197876158 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 338190542 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14077221157832 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚323308244896256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14076997508476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337672095742464 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 trainer/global_step‚7599 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14363850846208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140340082 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚301304978413056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51220981 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 $ _timestamp‚1.788423902640116e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14363577843712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚301304978413056 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14117978222080 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133329000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚299382038617032 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚299381797928316 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚301304996416000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14117978222080 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133328374 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285304817459200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚301304996416000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197850462 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338216240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14118206438912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚323308220956160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚315423202854912 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51220357 _step‚624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 338190542 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197876158 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140340080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82108017 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824ðz$õ2ž - _wandbruntime‚7373  _runtime‚7373iÜã)ZŸ Ô /loss/FRt1/detection_loss‚2.1761551753238395 0loss/FBt4/assignment_loss‚0.2593010252167005 epoch‚6 1loss/SRqqt3/assignment_loss‚1.443122399569142 1loss/SRqqt3/detection_loss‚1.6825252972981508 _runtime‚7373.392074756 2loss/FBt3/assignment_loss‚0.017621825356036423 /loss/FRt4/detection_loss‚2.0308373231963803 /loss/FRt4/assignment_loss‚2.601995498635048 2loss/SRqqt1/assignment_loss‚1.9331923111370115 /loss/FRt2/detection_loss‚1.5735002278039854 /loss/FRt3/detection_loss‚2.5936723195016382 1loss/SRqqt2/detection_loss‚0.7438584237474285 % _timestamp‚1.7884239026421924e+09 2loss/SRqqt2/assignment_loss‚0.8851893282661291 2loss/SRqqt4/assignment_loss‚1.3466379047267967 /loss/FBt3/detection_loss‚3.3534960985183715 1loss/FBt2/assignment_loss‚0.16042789981670047 .loss/FBt4/detection_loss‚2.243125692009926 _step‚625 0loss/FRt1/assignment_loss‚3.6402668301705963 .loss/FRt2/assignment_loss‚1.00425174087286 /loss/FRt3/assignment_loss‚3.211106465756893 /loss/FBt1/assignment_loss‚0.777577221315975 0loss/SRqqt1/detection_loss‚2.285473361159816 &loss/total_loss‚41.227231349575305 1loss/SRqqt4/detection_loss‚1.8765407118532393 .loss/FBt1/detection_loss‚2.774416317542394 /loss/FBt2/detection_loss‚0.6129399507802415 trainer/global_step‚7599>½¸é_  Ù /loss/FRt4/assignment_loss‚2.601995498635048 2loss/SRqqt1/assignment_loss‚1.9331923111370115 /loss/FRt2/detection_loss‚1.5735002278039854 /loss/FRt3/detection_loss‚2.5936723195016382 1loss/SRqqt2/detection_loss‚0.7438584237474285 % _timestamp‚1.7884239026421924e+09 2loss/SRqqt2/assignment_loss‚0.8851893282661291 2loss/SRqqt4/assignment_loss‚1.3466379047267967 /loss/FBt3/detection_loss‚3.3534960985183715 1loss/FBt2/assignment_loss‚0.16042789981670047 .loss/FBt4/detection_loss‚2.243125692009926 _step‚625 0loss/FRt1/assignment_loss‚3.6402668301705963 .loss/FRt2/assignment_loss‚1.00425174087286 /loss/FRt3/assignment_loss‚3.211106465756893 /loss/FBt1/assignment_loss‚0.777577221315975 0loss/SRqqt1/detection_loss‚2.285473361159816 &loss/total_loss‚41.227231349575305 1loss/SRqqt4/detection_loss‚1.8765407118532393 .loss/FBt1/detection_loss‚2.774416317542394 /loss/FBt2/detection_loss‚0.6129399507802415 trainer/global_step‚7599 /loss/FRt1/detection_loss‚2.1761551753238395 0loss/FBt4/assignment_loss‚0.2593010252167005 epoch‚6 1loss/SRqqt3/assignment_loss‚1.443122399569142 1loss/SRqqt3/detection_loss‚1.6825252972981508 _runtime‚7373.392074756 2loss/FBt3/assignment_loss‚0.017621825356036423 /loss/FRt4/detection_loss‚2.0308373231963803ñõ|U¡ ‚(:„ åÝäÔÔ–«x gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~œ"›i¢ ‚(:Þ åÝäÔ…«Ï{ network.sent‚ 1.8627331e+07 network.recv‚ 2.566521e+06 disk.rbd2.out‚ 222.638671875% disk.nvme0c0n1.out‚33320.85546875 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125$ memory_percent‚97.95210361480713 disk.rbd0.in‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.403101921081543 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.sda3.out‚ 369.5703125 disk.dm-0.in‚5.3125( disk./.usagePercent‚73.7157735335233% disk./.usageGB‚1098.4182968139648 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda.out‚ 369.609375 disk.sda2.in‚0 disk.dm-1.out‚ 369.5703125 disk.sda2.out‚ 0.0390625# proc.memory.rssMB‚ 2851.32421875 disk.rbd4.in‚0( proc.memory.availableMB‚ 335.52734375 cpu‚25.29474521888509 disk.sda1.out‚0 disk.nvme0n1.in‚5.3125 disk.dm-0.out‚33320.85546875# disk.nvme0n1.out‚33320.85546875 disk.rbd1.out‚ 3.74267578125 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17³Î+Nš£ ‚(: ôÝäÔêå²x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.57' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.913999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nTfÓr¤ ‚(:ç ôÝäÔ±š‘{ disk.sda2.out‚ 0.0390625 disk.sda1.out‚0 cpu‚25.294253958204163 disk.rbd3.in‚0 disk.sda1.in‚ 0.05078125# proc.memory.rssMB‚ 2851.32421875& disk.nvme0n1.out‚33321.23583984375 disk.rbd0.in‚0( disk./.usagePercent‚73.7157735335233 disk.dm-0.in‚5.3125$ memory_percent‚97.95143604278564 network.sent‚ 1.8735022e+07 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.403101921081543 disk.sda.out‚ 370.203125 disk.rbd2.out‚ 222.654296875 disk.nvme0c0n1.in‚5.3125 disk.rbd4.out‚0% disk./.usageGB‚1098.4182968139648 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.74267578125 disk.nvme0n1.in‚5.3125 disk.rbd4.in‚0( proc.memory.availableMB‚ 335.63671875 network.recv‚ 2.57253e+06 disk.sda3.out‚ 370.1640625 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 370.1640625 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚33321.23583984375 proc.cpu.threads‚17 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125# disk.dm-0.out‚33321.23583984375 disk.sda2.in‚0¥@2¥ ‚(:„ ƒÞäÔ§­x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.572' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9144 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Ô4p¦ ‚(:å ƒÞäÔ¥îòz disk.sda1.in‚ 0.05078125$ disk.nvme0n1.out‚33420.927734375 disk.sda2.in‚0* proc.memory.percent‚17.403101921081543 disk.sda3.in‚7.3125 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0) disk./.usagePercent‚73.71578658990755& disk.nvme0c0n1.out‚33420.927734375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2851.32421875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd1.in‚0! disk.dm-0.out‚33420.927734375' proc.memory.availableMB‚ 335.3984375 network.sent‚ 1.8741689e+07 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.rbd1.out‚ 3.74267578125 disk.drbd1777.in‚ 1.03515625 disk.sda3.out‚ 370.41796875$ memory_percent‚97.95289039611816 disk.sda.out‚ 370.45703125 network.recv‚ 2.578262e+06% disk./.usageGB‚1098.4184913635254 disk.dm-0.in‚5.3125 disk.dm-1.in‚7.3125 disk.rbd2.out‚ 222.716796875 disk.rbd4.out‚0 disk.rbd0.out‚0 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 370.41796875 cpu‚25.29379655722118 disk.rbd3.out‚0Äé{Ì2§ - _wandbruntime‚7418  _runtime‚7418jš™yƨ À & lr-AdamW/pg1‚0.0002869935115937093 & lr-AdamW/pg2‚0.0002869935115937093 trainer/global_step‚7649 $ _timestamp‚1.788423947645851e+09 _runtime‚7418.395806819 _step‚626’T\Ë© Å & lr-AdamW/pg1‚0.0002869935115937093 & lr-AdamW/pg2‚0.0002869935115937093 trainer/global_step‚7649 $ _timestamp‚1.788423947645851e+09 _runtime‚7418.395806819 _step‚626òF7²2ª - _wandbruntime‚7418  _runtime‚7418k&Æí'« çO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14455353664512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141208313 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚7418.396798417 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14208267868160 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14166798680084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚303183537672192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339781186693120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301250419534868 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚303183537672192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14455626680320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚317391823543296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82615414 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚317391577336832 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14208039664640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚339780889737216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 % _timestamp‚1.7884239476466713e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141208311 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199105813 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287083620854784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141208311 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199105813 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199131483 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚303183555675136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚317391577336832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚303183555675136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51538326 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 340314124 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301250660223376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 199131483 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134154376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚325325560012800 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 340339796 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚325325536072704 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 _step‚627 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82615412 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428‹va~—¬ ìO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚317391823543296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 trainer/global_step‚7649 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚287083637894144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚317391577336832 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14208039664640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚339780889737216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 % _timestamp‚1.7884239476466713e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141208311 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199105813 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287083620854784 GADeviceStatsMonitor.on_train_batch_start/active.large_p!œ¢¾[ ool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141208311 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199105813 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199131483 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚303183555675136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚317391577336832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚303183555675136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51538326 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 340314124 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301250660223376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 199131483 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134154376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚325325560012800 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 340339796 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚325325536072704 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 _step‚627 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82615412 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14455353664512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141208313 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚7418.396798417 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14208267868160 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14166798680084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚303183537672192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339781186693120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301250419534868 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚303183537672192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14455626680320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚317391823543296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82615414 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340357394 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 199157060 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199157060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 _runtime‚7419.294939032 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340383092 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141226030 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14168854729040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14209877359104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14168631079684 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14210105575936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340357394 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339824221204480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339823920067584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚287119923312640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134171480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚628 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301288554392324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚287119940352000 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141226032 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134170850 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚325366729709056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚301288795081040 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚317431752975360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚303221875616256 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141226032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14457214298624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51545083 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199131364 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340383092 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141226030 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14168854729040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14209877359104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14168631079684 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14210105575936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340357394 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339824221204480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339823920067584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚287119923312640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134171480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚628 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301288554392324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚287119940352000 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141226032 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134170850 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚325366729709056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚301288795081040 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚317431752975360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚303221875616256 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141226032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14457214298624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51545083 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199131364 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340357394 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 199157060 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199157060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628ô’ÿÎ\2° - _wandbruntime‚7419  _runtime‚7419_¬¸›[± Õ 0loss/FRt3/assignment_loss‚2.8922330210375233 2loss/SRqqt2/assignment_loss‚0.9342106118387843 2loss/SRqqt3/assignment_loss‚1.4544676819443703 1loss/SRqqt2/detection_loss‚1.0253584471420716 0loss/FRt2/assignment_loss‚1.3458434206540468 /loss/FRt4/assignment_loss‚3.135742885725839 0loss/SRqqt1/detection_loss‚2.285858230713086 -loss/FBt1/detection_loss‚2.09640606575542 _step‚629 /loss/FRt1/detection_loss‚2.8903811778873205 /loss/FRt4/detection_loss‚2.2478012868460757 /loss/FBt2/detection_loss‚0.5750871188222215 _runtime‚7419.296281582 2loss/SRqqt4/assignment_loss‚1.5836557975182166 .loss/FBt3/detection_loss‚2.175218254327774 /loss/FBt4/detection_loss‚2.3171321153640747 %loss/total_loss‚40.99735171720856 trainer/global_step‚7649 /loss/FRt1/assignment_loss‚4.228929590433836 1loss/SRqqt3/detection_loss‚2.0375367432832716 % _timestamp‚1.7884239485464394e+09 1loss/FBt4/assignment_loss‚0.12056617392227054 /loss/FRt3/detection_loss‚2.3489420268424723 epoch‚6 1loss/SRqqt1/assignment_loss‚1.566517105469337 1loss/FBt1/assignment_loss‚0.28383045612523955 1loss/FBt2/assignment_loss‚0.05398474115447563 0loss/FBt3/assignment_loss‚0.0744252410562088 /loss/FRt2/detection_loss‚1.5639119522004832 1loss/SRqqt4/detection_loss‚1.7593115711441407Óϼ!`² Ú 2loss/SRqqt4/assignment_loss‚1.5836557975182166 .loss/FBt3/detection_loss‚2.175218254327774 /loss/FBt4/detection_loss‚2.3171321153640747 %loss/total_loss‚40.99735171720856 trainer/global_step‚7649 /loss/FRt1/assignment_loss‚4.228929590433836 1loss/SRqqt3/detection_loss‚2.0375367432832716 % _timestamp‚1.7884239485464394e+09 1loss/FBt4/assignment_loss‚0.12056617392227054 /loss/FRt3/detection_loss‚2.3489420268424723 epoch‚6 1loss/SRqqt1/assignment_loss‚1.566517105469337 1loss/FBt1/assignment_loss‚0.28383045612523955 1loss/FBt2/assignment_loss‚0.05398474115447563 0loss/FBt3/assignment_loss‚0.0744252410562088 /loss/FRt2/detection_loss‚1.5639119522004832 1loss/SRqqt4/detection_loss‚1.7593115711441407 0loss/FRt3/assignment_loss‚2.8922330210375233 2loss/SRqqt2/assignment_loss‚0.9342106118387843 2loss/SRqqt3/assignment_loss‚1.4544676819443703 1loss/SRqqt2/detection_loss‚1.0253584471420716 0loss/FRt2/assignment_loss‚1.3458434206540468 /loss/FRt4/assignment_loss‚3.135742885725839 0loss/SRqqt1/detection_loss‚2.285858230713086 -loss/FBt1/detection_loss‚2.09640606575542 _step‚629 /loss/FRt1/detection_loss‚2.8903811778873205 /loss/FRt4/detection_loss‚2.2478012868460757 /loss/FBt2/detection_loss‚0.5750871188222215 _runtime‚7419.296281582õ‘ìÈ™³ ‚(:Ž ’ÞäÔ…ùŠx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.751' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.750200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>Šði´ ‚(:Þ ’ÞäÔ¨³Æz disk.rbd3.out‚0$ disk.nvme0n1.out‚33520.033203125 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 224.244140625 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 371.12890625# memory_percent‚97.9522705078125 network.recv‚ 2.582651e+06) disk./.usagePercent‚73.71578658990755 disk.rbd1.in‚0! disk.dm-0.out‚33520.033203125# proc.memory.rssMB‚ 2851.32421875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 371.08984375 disk.rbd3.in‚0 disk.dm-1.in‚7.3125 disk.rbd1.out‚ 3.74267578125 disk.dm-0.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125* proc.memory.percent‚17.403101921081543% disk./.usageGB‚1098.4184913635254 disk.rbd0.out‚0 disk.sda3.in‚7.3125 cpu‚25.29327470449344 disk.dm-1.out‚ 371.08984375 network.sent‚ 1.8747968e+07& disk.nvme0c0n1.out‚33520.033203125! proc.memory.availableMB‚335.5 disk.nvme0n1.in‚5.3125 disk.drbd1777.out‚0‡“QMµ ‚(:„ ¡ÞäÔ£¶‘z gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memúž¯BoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚97.289' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.4578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž>m'r¶ ‚(:ç ¡ÞäÔÆ–’| disk.sda3.out‚ 371.609375 network.sent‚ 1.8863504e+07# disk.dm-0.out‚33619.21337890625 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625& disk.nvme0n1.out‚33619.21337890625$ memory_percent‚97.95691967010498# proc.memory.rssMB‚ 2851.32421875 network.recv‚ 2.604103e+06 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.rbd1.out‚ 3.74267578125$ disk./.usageGB‚1098.418613433838 disk.nvme0c0n1.in‚5.3125 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 371.6484375 disk.dm-0.in‚5.3125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚33619.21337890625 disk.sda1.out‚0 disk.nvme0n1.in‚5.3125 cpu‚25.292808243012026) disk./.usagePercent‚73.71579478214865 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 224.306640625 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125( proc.memory.availableMB‚ 334.73828125 disk.dm-1.out‚ 371.609375 disk.rbd4.in‚0* proc.memory.percent‚17.403101921081543©xm‹· ‚(:„ °ÞäÔŠŽªx gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.972' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5944 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½Á^Úp¸ ‚(:å °ÞäÔÇɇ{$ disk./.usageGB‚1098.419101715088# disk.dm-0.out‚33619.56494140625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0# proc.memory.rssMB‚ 2851.32421875 proc.cpu.threads‚17 network.recv‚ 2.607606e+06 disk.rbd1.in‚0 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 372.2734375 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 cpu‚25.292295583362502 disk.sda2.in‚0 disk.dm-1.out‚ 372.2734375 disk.rbd2.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.403101921081543) disk./.usagePercent‚73.71582755111304& disk.nvme0n1.out‚33619.56494140625( disk.nvme0c0n1.out‚33619.56494140625# memory_percent‚97.9539155960083 disk.sda2.out‚ 0.0390625 network.sent‚ 1.8870924e+07 disk.sda3.in‚7.3125 disk.rbd1.out‚ 3.74267578125 disk.rbd2.out‚ 224.322265625 disk.rbd0.out‚0( proc.memory.availableMB‚ 335.23046875 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.sda.out‚372.3125×7œ2¹ - _wandbruntime‚7464  _runtime‚7464»—=£Çº Á trainer/global_step‚7699 % _timestamp‚1.7884239940875297e+09 _runtime‚7464.837454185 _step‚630 & lr-AdamW/pg1‚0.0002814538775460443 & lr-AdamW/pg2‚0.0002814538775460443G,ÉÌ» Æ trainer/global_step‚7699 % _timestamp‚1.7884239940875297e+09 _runtime‚7464.837454185 _step‚630 & lr-AdamW/pg1‚0.0002814538775460443 & lr-AdamW/pg2‚0.0002814538775460443öŒNœC2¼ - _wandbruntime‚7464  _runtime‚7464„¦”í'½ çO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚303157175998876 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51863448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14258655900440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200412385 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142094263 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200386715 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14300167005184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142094261 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14299938801664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83133162 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14548828441600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14258432251292 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305100434875392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚319400619883520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342480976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 % _timestamp‚1.7884239940882535e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51864109 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚319400619883520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305100434875392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327384044825600 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200412385 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14300167005184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚303157416687384 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚319400373677056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327384020885504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚288898760786944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚305100452878336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚319400373677056 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134997273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305100452878336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚7464.838395579 _step‚631 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142094261 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14299938801664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200386715 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚341932849327104 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83133162 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14548828441600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14258432251292 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305100434875392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚319400619883520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342480976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 % _timestamp‚1.7884239940882535e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51864109 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚319400619883520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305100434875392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327384044825600 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200412385 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14300167005184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚303157416687384 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚319400373677056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327384020885504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚288898760786944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚305100452878336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚319400373677056 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134997273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305100452878336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚7464.838395579 _step‚631 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142094261 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14299938801664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200386715 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚341932849327104 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142094261 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14299938801664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341975884613120 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚ 7465.74065345 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 trainer/global_step‚7699 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142111980 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚288935063244800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 % _timestamp‚1.7884239949899702e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327425214521856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135013328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14550694031360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14302004712960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327425190581760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _step‚632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342549944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚319440795535360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342549944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚288935046205440 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14260264650892 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83143517 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14301776496128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?Š{sDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚303195310856332 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚319440795535360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200437962 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142111980 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200412266 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305138772819456 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14301776496128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚319440549315584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342524246 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0¹¯Ÿ²û&Á õM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135013328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14550694031360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14302004712960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327425190581760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _step‚632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342549944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚319440795535360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342549944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚288935046205440 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14260264650892 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83143517 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14301776496128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚303195310856332 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚319440795535360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200437962 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142111980 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200412266 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305138772819456 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14301776496128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚319440549315584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342524246 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142111982 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200437962 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341975884613120 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚ 7465.74065345 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 trainer/global_step‚7699 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142111980 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚288935063244800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 % _timestamp‚1.7884239949899702e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327425214521856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ø[ßô2 - _wandbruntime‚7465  _runtime‚7465·ÑŸ¿Yà Ó 1loss/FBt3/assignment_loss‚0.15986578760202974 % _timestamp‚1.7884239949920323e+09 _step‚633 0loss/FRt2/assignment_loss‚1.3693043254969413 0loss/FBt2/detection_loss‚0.49009842704981565 2loss/SRqqt2/assignment_loss‚1.0735959786775142 0loss/FBt1/assignment_loss‚0.2690122686714555 0loss/SRqqt2/detection_loss‚0.933384755798788 /loss/FBt1/detection_loss‚2.9397144979900784 /loss/FRt3/detection_loss‚2.6130591481924057 /loss/FRt4/detection_loss‚2.2194164178930156 1loss/SRqqt1/detection_loss‚2.1485610110003774 2loss/SRqqt1/assignment_loss‚1.8885591891480655 /loss/FRt2/detection_loss‚1.3521321832872273 epoch‚6 trainer/global_step‚7699 /loss/FRt1/detection_loss‚2.9440904509376837 /loss/FRt1/assignment_loss‚3.609192740272831 /loss/FRt4/assignment_loss‚3.475999746567164 1loss/SRqqt3/assignment_loss‚1.288081129326632 1loss/SRqqt4/assignment_loss‚1.396293151743558 0loss/FBt2/assignment_loss‚0.0982388153571363 1loss/SRqqt4/detection_loss‚1.7240980906145913 .loss/FBt3/detection_loss‚2.411492254998949 .loss/FBt4/detection_loss‚4.119263708591461 %loss/total_loss‚44.16899558525066 _runtime‚7465.741874491 /loss/FRt3/assignment_loss‚3.274382279978858 1loss/FBt4/assignment_loss‚0.15648820168765573 0loss/SRqqt3/detection_loss‚2.214671024366429”Θo^Ä Ø _runtime‚7465.741874491 /loss/FRt3/assignment_loss‚3.274382279978858 1loss/FBt4/assignment_loss‚0.15648820168765573 0loss/SRqqt3/detection_loss‚2.214671024366429 1loss/FBt3/assignment_loss‚0.15986578760202974 % _timestamp‚1.7884239949920323e+09 _step‚633 0loss/FRt2/assignment_loss‚1.3693043254969413 0loss/FBt2/detection_loss‚0.49009842704981565 2loss/SRqqt2/assignment_loss‚1.0735959786775142 0loss/FBt1/assignment_loss‚0.2690122686714555 0loss/SRqqt2/detection_loss‚0.933384755798788 /loss/FBt1/detection_loss‚2.9397144979900784 /loss/FRt3/detection_loss‚2.6130591481924057 /loss/FRt4/detection_loss‚2.2194164178930156 1loss/SRqqt1/detection_loss‚2.1485610110003774 2loss/SRqqt1/assignment_loss‚1.8885591891480655 /loss/FRt2/detection_loss‚1.3521321832872273 epoch‚6 trainer/global_step‚7699 /loss/FRt1/detection_loss‚2.9440904509376837 /loss/FRt1/assignment_loss‚3.609192740272831 /loss/FRt4/assignment_loss‚3.475999746567164 1loss/SRqqt3/assignment_loss‚1.288081129326632 1loss/SRqqt4/assignment_loss‚1.396293151743558 0loss/FBt2/assignment_loss‚0.0982388153571363 1loss/SRqqt4/detection_loss‚1.7240980906145913 .loss/FBt3/detection_loss‚2.411492254998949 .loss/FBt4/detection_loss‚4.119263708591461 %loss/total_loss‚44.16899558525066ù#)•ÖŽÅ ‚(:ƒ ¿ÞäÔß±Âx gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡Í+npÆ ‚(:å ¿ÞäÔÄ¥Ë{ network.recv‚ 2.626069e+06( disk.nvme0c0n1.out‚33718.78662109375$ memory_percent‚97.95451164245605* proc.memory.percent‚17.403101921081543 disk.rbd0.in‚0$ disk./.usageGB‚1098.419101715088 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.rbd1.out‚ 3.74267578125# disk.dm-0.out‚33718.78662109375 disk.sda2.out‚ 0.0390625 network.sent‚ 1.8881575e+07 disk.rbd2.out‚224.4248046875 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.in‚0' proc.memory.availableMB‚ 335.1328125 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 proc.cpu.threads‚17 disk.sda3.out‚ 372.65625& disk.nvme0n1.out‚33718.78662109375 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.rbd4.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚73.71582755111304 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda.out‚ 372.6953125 disk.rbd3.in‚0 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 cpu‚25.291807082365036 disk.dm-1.out‚ 372.65625# proc.memory.rssMB‚ 2851.32421875²4VšŽÇ ‚(:ƒ ÎÞäÔÞÒ‹x gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ÈÛanÈ ‚(:ã ÎÞäÔÊ‹Á{ disk.sda.in‚ 7.36328125# proc.memory.rssMB‚ 2851.32421875 cpu‚25.29135809909259 network.sent‚ 1.899347e+07 disk.dm-1.out‚ 373.203125" disk.dm-0.out‚33817.9423828125 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚5.3125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚33817.9423828125 disk.rbd1.out‚ 3.74267578125) disk./.usagePercent‚73.71582755111304 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚ 373.2421875 disk.sda2.in‚0 disk.dm-1.in‚7.3125$ memory_percent‚97.95305728912354 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0( proc.memory.availableMB‚ 335.37109375% disk.nvme0n1.out‚33817.9423828125 network.recv‚ 2.639651e+06$ disk./.usageGB‚1098.419101715088 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125 disk.rbd2.out‚224.4404296875 proc.cpu.threads‚17 disk.dm-0.in‚5.3125 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 disk.sda3.out‚ 373.203125 disk.rbd1.in‚0* proc.memory.percent‚17.403101921081543žÃ°ùÉ ‚(:… ÝÞäÔ„¼€z gpu.0.gpu‚19.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.584' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?”ïÚsÊ ‚(:è ÝÞäÔ«ì©| disk.rbd3.out‚0 disk.sda.in‚ 7.36328125# proc.memory.rssMB‚ 2851.32421875 network.sent‚ 1.8999858e+07 disk.rbd2.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.403101921081543) disk./.usagePercent‚73.71582755111304 disk.sda2.in‚0 disk.drbd1777.out‚0$ disk./.usageGB‚1098.419101715088 disk.rbd1.in‚0 disk.rbd1.out‚ 3.74267578125 disk.dm-0.in‚5.3125 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.sda.out‚ 373.7421875 network.recv‚ 2.644608e+06 disk.nvme0n1.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚33917.30517578125 disk.sda1.in‚ 0.05078125 proc.cpu.threads‚17 disk.sda3.in‚7.3125# disk.dm-0.out‚33917.30517578125 disk.dm-1.out‚ 373.703125 disk.rbd0.out‚0( proc.memory.availableMB‚ 335.62109375 cpu‚25.290871329950043 disk.sda3.out‚ 373.703125& disk.nvme0n1.out‚33917.30517578125 disk.rbd4.in‚0$ memory_percent‚97.95153141021729 disk.rbd4.out‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚226.0771484375 disk.rbd3.in‚0~¬‡O2Ë - _wandbruntime‚7511  _runtime‚7511».J›ÆÌ À trainer/global_step‚7749 $ _timestamp‚1.788424040325884e+09 _runtime‚7511.075955651 _step‚634 & lr-AdamW/pg1‚0.0002759391899364039 & lr-AdamW/pg2‚0.0002759391899364039=XøÇËÍ Å & lr-AdamW/pg1‚0.0002759391899364039 & lr-AdamW/pg2‚0.0002759391899364039 trainer/global_step‚7749 $ _timestamp‚1.788424040325884e+09 _runtime‚7511.075955651 _step‚634úâ5,2Î - _wandbruntime‚7511  _runtime‚7511©qjïí'Ï çO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307017332078592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142980211 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52191190 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142980211 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290713883679744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142980213 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83650912 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14642305694720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚321409416223744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚307017332078592 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201667617 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14642582904832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321409416223744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52190556 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 344647828 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321409170017280 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307017350081536 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201693287 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135842104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚290713866640384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344673500 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚305063932462884 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884240403269005e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307017350081536 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14350289471648 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 _step‚635 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201667617 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14391837938688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14350065822500 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321409170017280 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14392066142208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14392066142208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚329442529638400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135841468 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142980213 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83650914ŠÊm—Á Ð ìO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚290713866640384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344673500 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚305063932462884 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884240403269005e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307017350081536 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14350289471648 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 _step‚635 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201667617 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14391837938688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14350065822500 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321409170017280 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14392066142208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14392066142208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚329442529638400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135841468 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142980213 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83650914 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307017332078592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142980211 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52191190 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142980211 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290713883679744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142980213 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83650912 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14642305694720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚321409416223744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚307017332078592 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201667617 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14642582904832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321409416223744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52190556 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 344647828 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321409170017280 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307017350081536 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201693287 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135842104ûzE³2Ñ - _wandbruntime‚7511  _runtime‚7511‡gˆ÷&Ò ñM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52197420 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290750169098240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344716796 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344691098 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚7511.961206715 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344127843485696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321449591875584 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135858689 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83661267 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14351898222100 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14644168091136 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135858074 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201693168 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344716796 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142997930 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14393903849984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚307055670022656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 _step‚636 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329483699334656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚329483675394560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305102308009056 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344128146719744 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201718864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321449591875584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14352121871456 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 201718864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344691098 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚307055688025600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880cN”ü&Ó öM YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14351898222100 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14644168091136 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135858074 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201693168 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344716796 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142997930 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14393903849984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚307055670022656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 _step‚636 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329483699334656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚329483675394560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305102308009056 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344128146719744 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201718864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321449591875584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14352121871456 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 201718864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344691098 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚307055688025600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52197420 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290750169098240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344716796 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344691098 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚7511.961206715 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344127843485696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321449591875584 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135858689 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83661267 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096üæܵl2Ô - _wandbruntime‚7511  _runtime‚7511ì `Õ Ú 0loss/FRt1/assignment_loss‚2.9343175436999345 /loss/FRt4/assignment_loss‚3.339597408260618 .loss/FRt2/detection_loss‚1.383891528897118 1loss/SRqqt2/detection_loss‚1.3790674541766446 /loss/FBt1/detection_loss‚2.9967651844024656 % _timestamp‚1.7884240412125273e+09 0loss/FBt3/assignment_loss‚0.8528588845466192 epoch‚6 _step‚637 1loss/FBt4/assignment_loss‚0.11919876816682518 /loss/FRt1/detection_loss‚2.5605551294378333 /loss/FRt3/detection_loss‚2.4830118247440884 /loss/FBt3/detection_loss‚2.1510499715805054 trainer/global_step‚7749 0loss/FRt2/assignment_loss‚1.2638827886497765 2loss/SRqqt1/assignment_loss‚1.6739617461158383 2loss/SRqqt4/assignment_loss‚1.3812684898867327 2loss/SRqqt3/assignment_loss‚1.5307651059519427 1loss/FBt2/assignment_loss‚0.12309824243669186 2loss/FBt1/assignment_loss‚0.007512793605292245 /loss/FRt4/detection_loss‚2.2044210443236034 0loss/FRt3/assignment_loss‚2.8849477626028515 2loss/SRqqt2/assignment_loss‚1.1302170944788184 0loss/SRqqt1/detection_loss‚2.490411854559375 /loss/FBt2/detection_loss‚0.6868572596824546 1loss/SRqqt3/detection_loss‚1.8314774422084583 1loss/SRqqt4/detection_loss‚1.8834829792088152 /loss/FBt4/detection_loss‚2.7149214914866855 %loss/total_loss‚42.00753979310999 _runtime‚7511.962373143à'{,eÖ ß 0loss/FBt3/assignment_loss‚0.8528588845466192 epoch‚6 _step‚637 1loss/FBt4/assignment_loss‚0.11919876816682518 /loss/FRt1/detection_loss‚2.5605551294378333 /loss/FRt3/detection_loss‚2.4830118247440884 /loss/FBt3/detection_loss‚2.1510499715805054 trainer/global_step‚7749 0loss/FRt2/assignment_loss‚1.2638827886497765 2loss/SRqqt1/assignment_loss‚1.6739617461158383 2loss/SRqqt4/assignment_loss‚1.3812684898867327 2loss/SRqqt3/assignment_loss‚1.5307651059519427 1loss/FBt2/assignment_loss‚0.12309824243669186 2loss/FBt1/assignment_loss‚0.007512793605292245 /loss/FRt4/detection_loss‚2.2044210443236034 0loss/FRt3/assignment_loss‚2.8849477626028515 2loss/SRqqt2/assignment_loss‚1.1302170944788184 0loss/SRqqt1/detection_loss‚2.490411854559375 /loss/FBt2/detection_loss‚0.6868572596824546 1loss/SRqqt3/detection_loss‚1.8314774422084583 1loss/SRqqt4/detection_loss‚1.8834829792088152 /loss/FBt4/detection_loss‚2.7149214914866855 %loss/total_loss‚42.00753979310999 _runtime‚7511.962373143 0loss/FRt1/assignment_loss‚2.9343175436999345 /loss/FRt4/assignment_loss‚3.339597408260618 .loss/FRt2/detection_loss‚1.383891528897118 1loss/SRqqt2/detection_loss‚1.3790674541766446 /loss/FBt1/detection_loss‚2.9967651844024656 % _timestamp‚1.7884240412125273e+09ýÌ“›× ‚(:… ìÞäÔƒ†›x gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.917' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ빿xØ ‚(:í ìÞäÔœ©Ÿ{ disk.sda.in‚ 7.36328125 disk.dm-1.out‚ 374.32421875 disk.rbd2.in‚0# proc.memory.rssMB‚ 2851.32421875% disk./.usageGB‚1098.4189376831055 disk.sda3.out‚ 374.32421875 disk.rbd1.in‚0 disk.dm-1.in‚7.3125 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 disk.sda.out‚ 374.36328125 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.sda2.in‚0$ memory_percent‚97.95317649841309 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125' proc.memory.availableMB‚ 335.3515625 disk.sda3.in‚7.3125 disk.rbd2.out‚226.0966796875 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.74267578125& disk.nvme0n1.out‚33917.75244140625 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 cpu‚25.290399726010275 proc.cpu.threads‚17 disk.rbd3.out‚0( disk.nvme0c0n1.out‚33917.75244140625 disk.rbd3.in‚0# disk.dm-0.out‚33917.75244140625 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0* proc.memory.percent‚17.403101921081543 network.sent‚ 1.9005188e+07 network.recv‚ 2.648095e+06) disk./.usagePercent‚73.71581654278906þu1¶Ù ‚(:… ûÞäÔøø¯x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðåÍÐÎÚ ‚(:ê ûÞäÔ²ã{ disk.rbd3.out‚0 disk.rbd1.in‚0* proc.memory.percent‚17.403101921081543 network.recv‚ 2.655282e+06 disk.dm-1.out‚ 374.71484375 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚0' proc.memory.availableMB‚ 335.5859375) disk./.usagePercent‚73.71581654278906 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.nvme0n1.in‚5.3125% disk.nvme0n1.out‚34017.1845703125 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda3.in‚7.3125# proc.memory.rssMB‚ 2851.32421875 network.sent‚ 1.9113611e+07 proc.cpu.threads‚17 disk.dm-0.in‚5.3125 disk.dm-1.in‚7.3125 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0$ memory_percent‚97.95174598693848 disk.sda.in‚ 7.36328125 disk.sda.out‚ 374.75390625% disk./.usageGB‚1098.4189376831055 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd2.out‚226.1591796875" disk.dm-0.out‚3401{pgN§7.1845703125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚34017.1845703125 disk.sda3.out‚ 374.71484375 disk.rbd1.out‚ 3.74267578125 cpu‚25.289918924148807 tĆ›Û ‚(: ŠßäÔ¾±˜x gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.467' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.293400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ùš¾uÜ ‚(:ê ŠßäÔ¥Õ·z$ memory_percent‚97.95351028442383 disk.dm-1.out‚ 375.26953125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚34116.40869140625 disk.sda.out‚ 375.30859375 disk.sda3.in‚7.3125 disk.rbd1.out‚ 3.74267578125 disk.dm-1.in‚7.3125 cpu‚25.28945285341974 network.recv‚ 2.65933e+06% disk./.usageGB‚1098.4186820983887* proc.memory.percent‚17.403101921081543 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 network.sent‚ 1.9119506e+07& disk.nvme0n1.out‚34116.40869140625 disk.sda2.in‚0 disk.drbd1777.out‚0& proc.memory.availableMB‚ 335.296875 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd4.out‚0) disk./.usagePercent‚73.71579939028426 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.dm-0.in‚5.3125# proc.memory.rssMB‚ 2851.32421875 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚34116.40869140625 disk.rbd2.out‚226.1748046875 disk.rbd4.in‚0 disk.sda3.out‚ 375.26953125 disk.rbd1.in‚0´gßY2Ý - _wandbruntime‚7557  _runtime‚7557ÒŠq†ÉÞ Ã % _timestamp‚1.7884240868713756e+09 _runtime‚7557.621317157 _step‚638 ' lr-AdamW/pg1‚0.00027045059801742056 ' lr-AdamW/pg2‚0.00027045059801742056 trainer/global_step‚7799vae£Îß È _runtime‚7557.621317157 _step‚638 ' lr-AdamW/pg1‚0.00027045059801742056 ' lr-AdamW/pg2‚0.00027045059801742056 trainer/global_step‚7799 % _timestamp‚1.7884240868713756e+09þðñ"U2à - _wandbruntime‚7557  _runtime‚7557ÓÜí'á çO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚308934229281792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14736067860480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚292528989533184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52514212 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143866161 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202974189 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143866161 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚306970688926892 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202974189 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323418212563968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143866163 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323418212563968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚308934247284736 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136682874 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 346840352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323417966357504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14483737075712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14441699393708 % _timestamp‚1.7884240868722427e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14483965279232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84168664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202948519 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _step‚639 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚308934247284736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202948519 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 trainer/global_step‚7799 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 346814680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143866163 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84168662 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚346237082311680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331501014451200 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331500990511104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14735796941824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14483737075712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14441923042856 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136683540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323417966357504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14483737075712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14441699393708 % _timestamp‚1.7884240868722427e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14483965279232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84168664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202948519 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _step‚639 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚308934247284736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202948519 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 trainer/global_step‚7799 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 346814680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143866163 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84168662 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚346237082311680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331501014451200 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331500990511104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14735796941824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14483737075712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14441923042856 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136683540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚308934229281792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14736067860480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚292528989533184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52514212 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143866161 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202974189 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143866161 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚306970688926892 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202974189 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323418212563968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143866163 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323418212563968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880ÿ4ë›ó2ã - _wandbruntime‚7558  _runtime‚75585ð^æð&ä ñM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84179019 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52520971 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚346279838993408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚331542160207360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323458141996032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14737955982848 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚292565309030400 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14485574770176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14443755442664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202999766 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308972585228800 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136699386 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 346883648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14443531793308 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚308972567225856 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202974070 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52520369 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚307009064473064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 346857950 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14737678786048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 trainer/global_step‚7799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143883882 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 346883648 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136699990 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143883880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331542184147456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308972585228800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14485802987008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 346857950 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143883880 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308972567225856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 % _timestamp‚1.7884240877547827e+09 _step‚640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202974070 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚604 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚292565291991040 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚à)ê25696[gü&å öM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚308972567225856 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202974070 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52520369 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚307009064473064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 346857950 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14737678786048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 trainer/global_step‚7799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143883882 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 346883648 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136699990 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143883880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331542184147456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308972585228800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14485802987008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 346857950 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143883880 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308972567225856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 % _timestamp‚1.7884240877547827e+09 _step‚640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202974070 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚604 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚292565291991040 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84179019 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52520971 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚346279838993408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚331542160207360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323458141996032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14737955982848 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚292565309030400 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14485574770176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14443755442664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202999766 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308972585228800 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136699386 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 346883648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14443531793308 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624€¨r0,2æ - _wandbruntime‚7558  _runtime‚75585DZç Ô 2loss/SRqqt2/assignment_loss‚0.9690698521083285 /loss/FRt1/detection_loss‚2.1763068842499154 .loss/FRt3/detection_loss‚2.388190197093146 /loss/FBt2/detection_loss‚0.6464097371014456 %loss/total_loss‚41.48353975476336 /loss/FRt1/assignment_loss‚3.356040138265361 0loss/SRqqt1/detection_loss‚2.125946138586317 2loss/SRqqt1/assignment_loss‚1.6658283120819501 .loss/FBt1/assignment_loss‚0.21651577098029 1loss/FBt3/assignment_loss‚0.14801849542224468 /loss/FRt2/detection_loss‚1.6684928691387177 /loss/FRt4/detection_loss‚2.1920306419507143 /loss/FRt3/assignment_loss‚3.030724380697523 1loss/SRqqt2/detection_loss‚0.9684777083196516 /loss/FBt3/detection_loss‚2.5022933615578546 % _timestamp‚1.7884240877565494e+09 2loss/SRqqt3/assignment_loss‚1.5090055871814332 0loss/FBt4/assignment_loss‚0.3508020759327337 1loss/SRqqt4/detection_loss‚2.1132925990968943 epoch‚6 trainer/global_step‚7799 0loss/FRt2/assignment_loss‚1.6870861554145813 1loss/FBt2/assignment_loss‚0.09130027998569258 .loss/FBt4/detection_loss‚2.650095507502556 0loss/FRt4/assignment_loss‚3.0251522387067475 _step‚641 2loss/SRqqt4/assignment_loss‚1.2698120482265949 1loss/SRqqt3/detection_loss‚1.9192634418289711 /loss/FBt1/detection_loss‚2.8133853333336964 _runtime‚7558.506372387 6Ç_è Ù 1loss/FBt2/assignment_loss‚0.09130027998569258 .loss/FBt4/detection_loss‚2.650095507502556 0loss/FRt4/assignment_loss‚3.0251522387067475 _step‚641 2loss/SRqqt4/assignment_loss‚1.2698120482265949 1loss/SRqqt3/detection_loss‚1.9192634418289711 /loss/FBt1/detection_loss‚2.8133853333336964 _runtime‚7558.506372387 2loss/SRqqt2/assignment_loss‚0.9690698521083285 /loss/FRt1/detection_loss‚2.1763068842499154 .loss/FRt3/detection_loss‚2.388190197093146 /loss/FBt2/detection_loss‚0.6464097371014456 %loss/total_loss‚41.48353975476336 /loss/FRt1/assignment_loss‚3.356040138265361 0loss/SRqqt1/detection_loss‚2.125946138586317 2loss/SRqqt1/assignment_loss‚1.6658283120819501 .loss/FBt1/assignment_loss‚0.21651577098029 1loss/FBt3/assignment_loss‚0.14801849542224468 /loss/FRt2/detection_loss‚1.6684928691387177 /loss/FRt4/detection_loss‚2.1920306419507143 /loss/FRt3/assignment_loss‚3.030724380697523 1loss/SRqqt2/detection_loss‚0.9684777083196516 /loss/FBt3/detection_loss‚2.5022933615578546 % _timestamp‚1.7884240877565494e+09 2loss/SRqqt3/assignment_loss‚1.5090055871814332 0loss/FBt4/assignment_loss‚0.3508020759327337 1loss/SRqqt4/detection_loss‚2.1132925990968943 epoch‚6 trainer/global_step‚7799 0loss/FRt2/assignment_loss‚1.6870861554145813v:3é ‚(:„ ™ßäÔÙ¥Œz gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚90.544' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1088 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 @È>kê ‚(:à ™ßäÔ³ÂÀ| disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 network.recv‚ 2.664325e+06) disk./.usagePercent‚73.71589948923015 disk.sda.in‚ 7.36328125( proc.memory.availableMB‚ 334.76171875! disk.dm-0.out‚34117.041015625 disk.sda2.out‚ 0.0390625 proc.cpu.threads‚17 disk.nvme0n1.in‚5.3125 disk.nvme0c0n1.in‚5.3125& disk.nvme0c0n1.out‚34117.041015625 disk.rbd3.in‚0 disk.sda3.in‚7.3125$ memory_percent‚97.95677661895752% disk./.usageGB‚1098.4201736450195 disk.dm-1.out‚375.8125 disk.rbd0.out‚0 disk.rbd4.out‚0$ disk.nvme0n1.out‚34117.041015625* proc.memory.percent‚17.403101921081543! disk.rbd2.out‚226.28076171875 disk.dm-0.in‚5.3125 cpu‚25.288965128297544 network.sent‚ 1.9125848e+07 disk.dm-1.in‚7.3125 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda3.out‚375.8125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 375.8515625 disk.rbd1.out‚ 3.74267578125# proc.memory.rssMB‚ 2851.32421875µˆîë ‚(:… ¨ßäÔ⊗x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.113' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾(÷rì ‚(:ç ¨ßäÔá³ëz disk.rbd4.in‚0% disk./.usageGB‚1098.4197006225586 disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.sda2.out‚ 0.0390625 network.recv‚ 2.669844e+06( disk./.usagePercent‚73.7158677442959 disk.nvme0c0n1.in‚5.3125 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.rbd4.out‚0$ memory_percent‚97.95475006103516 proc.cpu.threads‚17 disk.sda3.out‚ 376.3203125 disk.nvme0n1.in‚5.3125 disk.dm-1.out‚ 376.3203125 disk.drbd1777.in‚ 1.03515625% proc.memory.availableMB‚ 335.09375# proc.memory.rssMB‚ 2851.32421875 disk.sda1.out‚0 disk.drbd1777.out‚0 cpu‚25.288505055018142 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0! disk.rbd2.out‚226.28076171875 network.sent‚ 1.9232837e+07( disk.nvme0c0n1.out‚34216.19873046875 disk.rbd3.out‚0 disk.sda.out‚ 376.359375 disk.dm-0.in‚5.3125* proc.memory.percent‚17.403101921081543 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 3.74267578125& disk.nvme0n1.out‚34216.19873046875# disk.dm-0.out‚34216.19873046875è¡x›í ‚(: ·ßäÔÓ½“x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ Û‹yî ‚(:î ·ßäÔƒÓÙz disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.sda3.out‚ 376.75390625* proc.memory.percent‚17.403101921081543 disk.rbd3.in‚0 disk.rbd1.in‚0# disk.dm-0.out‚34315.74658203125 disk.drbd1777.out‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚34315.74658203125 network.sent‚ 1.9238822e+07% disk./.usageGB‚1098.4194145202637& disk.nvme0n1.out‚34315.74658203125 disk.dm-0.in‚5.3125 disk.sda.out‚ 376.79296875' proc.memory.availableMB‚ 334.8515625 cpu‚25.288036299413335 disk.sda1.out‚0$ memory_percent‚97.95622825622559 disk.nvme0n1.in‚5.3125 disk.dm-1.out‚ 376.75390625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2851.32421875 disk.rbd1.out‚ 3.74267578125 disk.sda2.out‚ 0.0390625 disk.sda1.in‚ 0.05078125 proc.cpu.threads‚17 disk.rbd2.in‚0! disk.rbd2.out‚227.99560546875 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0 network.recv‚ 2.674482e+06) disk./.usagePercent‚73.71584854373083!e2ï - _wandbruntime‚7604  _runtime‚76046´›Éð à % _timestamp‚1.7884241334186738e+09 _runtime‚7604.168626497 _step‚642 ' lr-AdamW/pg1‚0.00026498924560342365 ' lr-AdamW/pg2‚0.00026498924560342365 trainer/global_step‚7849PX,fÎñ È % _timestamp‚1.7884241334186738e+09 _runtime‚7604.168626497 _step‚642 ' lr-AdamW/pg1‚0.00026498924560342365 ' lr-AdamW/pg2‚0.00026498924560342365 trainer/global_step‚7849‚ã –õ2ò - _wandbruntime‚7604  _runtime‚7604kM!gí'ó çO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144752113 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14533556614064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14533332964916 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84686412 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14575636212736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294344129465344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137528704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚325427008904192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52842290 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333559499264000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚643 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325426762697728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310851126484992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 % _timestamp‚1.7884241334194486e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚308877445390900 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14829654720512 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14575864416256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52841626 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 _runtime‚7604.169578281 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349007204 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚308877686079408 trainer/global_step‚7849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14575864416256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14575636212736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204255091 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144752111 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144752111 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204255091 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137528038 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14829385899008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310851144487936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144752113 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310851144487936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348388861222912 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚325426762697728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚325427008904192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333559475323904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310851126484992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0'ˆs@£ô ìO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚325426762697728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚325427008904192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333559475323904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310851126484992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144752113 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14533556614064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14533332964916 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84686412 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14575636212736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294344129465344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137528704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚325427008904192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52842290 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333559499264000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚643 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325426762697728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310851126484992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 % _timestamp‚1.7884241334194486e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚308877445390900 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14829654720512 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14575864416256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52841626 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 _runtime‚7604.169578281 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349007204 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚308877686079408 trainer/global_step‚7849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14575864416256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14575636212736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204255091 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144752111 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144752111 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204255091 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137528038 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14829385899008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310851144487936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144752113 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310851144487936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348388861222912 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508ƒî±"[Žõ ‚(:ƒ ÆßäÔäâx gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.816 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K¥9oö ‚(:ä ÆßäÔïçÍz' disk.nvme0c0n1.out‚34415.0498046875 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.dm-0.in‚5.3125 disk.dm-1.in‚7.3125 cpu‚25.287569041171803 disk.sda1.out‚0 disk.dm-1.out‚ 377.5234375 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.drbd1777.in‚ 1.03515625# proc.memory.rssMB‚ 2851.32421875 proc.cpu.threads‚17 network.sent‚ 1.9244913e+07$ disk./.usageGB‚1098.419506072998 disk.rbd3.out‚0 disk.rbd0.in‚0* proc.memory.percent‚17.403101921081543 disk.rbd2.in‚0 disk.sda.out‚377.5625 disk.sda3.out‚ 377.5234375 disk.rbd1.out‚ 3.74267578125% disk.nvme0n1.out‚34415.0498046875' proc.memory.availableMB‚ 334.8515625 network.recv‚ 2.678603e+06 disk.sda3.in‚7.3125 disk.nvme0n1.in‚5.3125 disk.nvme0c0n1.in‚5.3125$ memory_percent‚97.95622825622559! disk.rbd2.out‚227.99560546875 disk.rbd4.in‚0) disk./.usagePercent‚73.71585468791166" disk.dm-0.out‚34415.0498046875 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd4.out‚0¯›]82÷ - _wandbruntime‚7605  _runtime‚7605àkIä÷&ø ñM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14831246640640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325466938336256 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52848392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚348432196992000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14535165364516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349050500 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204280668 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204254972 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310889482432000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144769830 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚308915820937072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333600645020160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294380414883840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚308915580248356 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 305331200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚310889464429056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349050500 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14535389013872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144769832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84696769 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14831528031744 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 204280668 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 % _timestamp‚1.7884241343135695e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 trainer/global_step‚7849 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325467184556032 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14577702124032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52847754 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144769830 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _step‚644 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204254972 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325467184556032 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137544521 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349024802 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310889464429056 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14577473907200 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14577473907200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310889482432000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128S¬åòü&ù öM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333600668960256 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84696767 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333600645020160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294380414883840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚308915580248356 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 305331200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚310889464429056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349050500 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14535389013872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144769832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84696769 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14831528031744 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 204280668 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 % _timestamp‚1.7884241343135695e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 trainer/global_step‚7849 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325467184556032 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14577702124032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52847754 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144769830 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _step‚644 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204254972 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325467184556032 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137544521 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349024802 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310889464429056 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14577473907200 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14577473907200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310889482432000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14831246640640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325466938336256 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52848392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚348432196992000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14535165364516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349050500 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204280668 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204254972 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310889482432000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144769830 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚308915820937072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F»€!‚(:ä äßäÔŸ¹Üz disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚34514.93896484375 disk.rbd1.out‚ 3.74267578125# disk.dm-0.out‚34514.93896484375( proc.memory.availableMB‚ 334.94921875 disk.rbd1.in‚0* proc.memory.percent‚17.403101921081543 cpu‚25.28663800184114' disk./.usagePercent‚73.715853407874 disk.sda3.out‚ 378.7578125 disk.dm-0.in‚5.3125 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 2.691827e+06 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.dm-1.out‚ 378.7578125 disk.rbd2.out‚ 228.1015625 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚7.,aü@´3125 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.sda2.out‚ 0.0390625% disk./.usageGB‚1098.4194869995117& disk.nvme0n1.out‚34514.93896484375# proc.memory.rssMB‚ 2851.32421875 disk.nvme0n1.in‚5.3125 network.sent‚ 1.9439484e+07 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 378.796875$ memory_percent‚97.955632209777838¢ý !‚(:„ óßäÔˆ†£x gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ{̓o‚!‚(:ä óßäÔ•Ä»{ network.recv‚ 2.695929e+06 disk.rbd1.out‚ 3.74267578125 disk.dm-0.in‚5.3125 disk.sda.in‚ 7.36328125) disk./.usagePercent‚73.69840163027236% disk./.usageGB‚1098.1594429016113 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0& disk.nvme0c0n1.out‚34614.802734375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.sda3.in‚7.3125! disk.dm-0.out‚34614.802734375$ memory_percent‚97.95737266540527* proc.memory.percent‚17.403101921081543 network.sent‚ 1.9445394e+07 disk.rbd4.in‚0$ disk.nvme0n1.out‚34614.802734375# proc.memory.rssMB‚ 2851.32421875 proc.cpu.threads‚17 disk.sda1.in‚ 0.05078125 cpu‚25.286137998145456 disk.rbd3.out‚0 disk.sda.out‚ 379.05078125 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 379.01171875 disk.rbd2.out‚ 228.2421875 disk.rbd3.in‚0 disk.sda3.out‚ 379.01171875' proc.memory.availableMB‚ 334.6640625ývƒ2ƒ!- _wandbruntime‚7650  _runtime‚7650·Û¾`È„! % _timestamp‚1.7884241792702887e+09 _runtime‚ 7650.02017612 _step‚646 ' lr-AdamW/pg1‚0.00025955627083207056 ' lr-AdamW/pg2‚0.00025955627083207056 trainer/global_step‚7899êÊYÍ…!Ç ' lr-AdamW/pg1‚0.00025955627083207056 ' lr-AdamW/pg2‚0.00025955627083207056 trainer/global_step‚7899 % _timestamp‚1.7884241792702887e+09 _runtime‚ 7650.02017612 _step‚646†aÝ\2†!- _wandbruntime‚7650  _runtime‚7650ÈYßí'‡!çO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 205510323 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350540784464384 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14667763553280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138368351 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 205535993 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85204164 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14922824327680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚296159235318784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53163557 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚310784442543416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327435559037952 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 351148384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14624966536124 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145638063 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312768023688192 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138367719 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14625190185272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 205535993 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚327435805244416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14667763553280 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145638063 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚296159252358144 trainer/global_step‚7899 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 _runtime‚7650.021203987 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327435805244416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350541083517440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312768041691136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327435559037952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚335617960136704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14667535349760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85204162 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145638061 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205510323 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚632 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312768041691136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327435559037952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚335617960136704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14667535349760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85204162 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145638061 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205510323 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚632 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145638063 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312768023688192 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138367719 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14625190185272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 205535993 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚327435805244416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14667763553280 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672‡„ºPç2‰!- _wandbruntime‚7650  _runtime‚7650Â3Á÷%Š!ðM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85214519 $ _timestamp‚1.788424180152912e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14924702190592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚296195537776640 trainer/global_step‚7899 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14669373044224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 205561570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53169793 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350583832023552 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14669601261056 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚327475734676480 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 351191654 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 _step‚648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 351191654 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312806379635200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14626798935724 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚312806361632256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351217352 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14627022585080 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145655780 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145655780 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚335659129832960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205535874 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 351217352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 305331200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚310822336712364 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚335659153773056 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14924983581696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14669373044224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚296195554816000 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138384310 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚310822577401080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53170423 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚312806361632256 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138384942 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350584137354752 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145655782 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85214517 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚327475980896256 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145655782 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inacti1rÞNÿve_split.small_pool.peak‚1243 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚312806361632256 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138384942 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350584137354752 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145655782 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85214517 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚327475980896256 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145655782 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚296195537776640 trainer/global_step‚7899 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14669373044224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 205561570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53169793 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350583832023552 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14669601261056 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚327475734676480 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 351191654 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 _step‚648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 351191654 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312806379635200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14626798935724 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚312806361632256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351217352 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14627022585080 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145655780 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145655780 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚335659129832960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205535874 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 351217352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 305331200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚310822336712364 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚335659153773056 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14924983581696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14669373044224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚296195554816000 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138384310 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚310822577401080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53170423 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2ˆ#û82Œ!- _wandbruntime‚7650  _runtime‚7650De€M]!× /loss/FRt3/assignment_loss‚3.094521254301071 /loss/FRt4/assignment_loss‚3.238901255082111 2loss/SRqqt2/assignment_loss‚0.8470743147523275 1loss/FBt3/assignment_loss‚0.35579650386353023 /loss/FRt1/detection_loss‚3.0184804377732455 /loss/FRt2/detection_loss‚1.3994447190699906 .loss/FBt1/detection_loss‚6.623932301998138 /loss/FBt4/detection_loss‚2.5651398400465646 trainer/global_step‚7899 1loss/FBt2/assignment_loss‚0.18795942096028836 0loss/FBt4/assignment_loss‚0.0108937545446679 1loss/SRqqt2/detection_loss‚0.9348068648150989 _runtime‚ 7650.90448881 0loss/FBt1/assignment_loss‚1.0370711088180542 _step‚649 /loss/FRt3/detection_loss‚2.7283756427074732 1loss/SRqqt4/detection_loss‚1.8212272986196554 /loss/FBt3/detection_loss‚2.6628644093871117 &loss/total_loss‚47.414738350911335 2loss/SRqqt1/assignment_loss‚1.9802407542864482 .loss/FRt4/detection_loss‚2.226503058654092 1loss/SRqqt3/detection_loss‚1.9453930508117288 /loss/FBt2/detection_loss‚0.6194068725123127 0loss/FRt1/assignment_loss‚3.3707980668103255 0loss/FRt2/assignment_loss‚1.2251802014893498 2loss/SRqqt3/assignment_loss‚1.5888081270517136 2loss/SRqqt4/assignment_loss‚1.4476040876112306 epoch‚6 % _timestamp‚1.7884241801546686e+09 1loss/SRqqt1/detection_loss‚2.4843150049448015d÷bŽ!Ü 0loss/FRt1/assignment_loss‚3.3707980668103255 0loss/FRt2/assignment_loss‚1.2251802014893498 2loss/SRqqt3/assignment_loss‚1.5888081270517136 2loss/SRqqt4/assignment_loss‚1.4476040876112306 epoch‚6 % _timestamp‚1.7884241801546686e+09 1loss/SRqqt1/detection_loss‚2.4843150049448015 /loss/FRt3/assignment_loss‚3.094521254301071 /loss/FRt4/assignment_loss‚3.238901255082111 2loss/SRqqt2/assignment_loss‚0.8470743147523275 1loss/FBt3/assignment_loss‚0.35579650386353023 /loss/FRt1/detection_loss‚3.0184804377732455 /loss/FRt2/detection_loss‚1.3994447190699906 .loss/FBt1/detection_loss‚6.623932301998138 /loss/FBt4/detection_loss‚2.5651398400465646 trainer/global_step‚7899 1loss/FBt2/assignment_loss‚0.18795942096028836 0loss/FBt4/assignment_loss‚0.0108937545446679 1loss/SRqqt2/detection_loss‚0.9348068648150989 _runtime‚ 7650.90448881 0loss/FBt1/assignment_loss‚1.0370711088180542 _step‚649 /loss/FRt3/detection_loss‚2.7283756427074732 1loss/SRqqt4/detection_loss‚1.8212272986196554 /loss/FBt3/detection_loss‚2.6628644093871117 &loss/total_loss‚47.414738350911335 2loss/SRqqt1/assignment_loss‚1.9802407542864482 .loss/FRt4/detection_loss‚2.226503058654092 1loss/SRqqt3/detection_loss‚1.9453930508117288 /loss/FBt2/detection_loss‚0.6194068725123127‰P³Ñ~!‚(:… ‚àäÔô‡¸x gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nE´q!‚(:æ ‚àäÔØòÇ{ disk.rbd2.out‚ 229.81640625 disk.nvme0c0n1.in‚5.3125 disk.dm-1.out‚ 379.57421875 disk.rbd4.out‚0# proc.memory.rssMB‚ 2851.32421875! disk.dm-0.out‚34781.330078125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.sda1.in‚ 0.05078125* proc.memory.percent‚17.403101921081543 cpu‚25.285676784610715 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda1.out‚0 network.recv‚ 2.699732e+06) disk./.usagePercent‚73.64458654250724% disk./.usageGB‚1097.3575592041016 disk.rbd1.out‚ 3.74267578125 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda.in‚ 7.36328125 disk.sda.out‚ 379.61328125$ memory_percent‚97.95606136322021( proc.memory.availableMB‚ 334.87890625 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.rbd3.out‚0$ disk.nvme0n1.out‚34781.330078125 disk.dm-0.in‚5.3125 disk.rbd0.in‚0& disk.nvme0c0n1.out‚34781.330078125 disk.sda3.out‚ 379.57421875 disk.rbd4.in‚0 network.sent‚ 1.9451253e+07 disk.nvme0n1.in‚5.3125RN±©‘!‚(:… ‘àäÔîœÉx gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏC¹´l’!‚(:á ‘àäÔÆ‚Æ{% disk./.usageGB‚1097.3575592041016 disk.rbd4.out‚0% disk.nvme0c0n1.out‚34782.26953125 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚ 380.04296875# memory_percent‚97.9560375213623' proc.memory.availableMB‚ 334.8828125 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 380.00390625 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2851.32421875* proc.memory.percent‚17.403101921081543 disk.rbd4.in‚0 disk.sda1.out‚0 network.sent‚ 1.9559967e+07) disk./.usagePercent‚73.64458654250724 disk.sda1.in‚ 0.05078125 disk.dm-0.out‚34782.26953125 cpu‚25.285185775740434 proc.cpu.threads‚17 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.rbd1.out‚ 3.74267578125 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.nvme0n1.in‚5.3125 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 380.00390625 disk.rbd2.out‚ 229.89453125 network.recv‚ 2.707326e+06# disk.nvme0n1.out‚34782.26953125“J™eŽ“!‚(:ƒ  àäÔõÜÖx gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.46' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.892 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÄ}Ur”!‚(:ç  àäÔûÇÛ{ disk.rbd1.out‚ 3.74267578125 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.in‚ 7.36328125 disk.dm-1.out‚ 380.76953125# memory_percent‚97.9560375213623 proc.cpu.threads‚17 network.sent‚ 1.956528e+07$ disk./.usageGB‚1097.357681274414 disk.drbd1777.in‚ 1.03515625 disk.sda3.out‚ 380.76953125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2851.32421875 cpu‚25.284760113345197 disk.rbd0.out‚0' proc.memory.availableMB‚ 334.8828125 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0# disk.dm-0.out‚34782.82763671875 disk.rbd2.out‚ 229.89453125 disk.rbd1.in‚0& disk.nvme0n1.out‚34782.82763671875) disk./.usagePercent‚73.64459473474834 disk.sda3.in‚7.3125 disk.dm-1.in‚7.3125 disk.rbd4.out‚0( disk.nvme0c0n1.out‚34782.82763671875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 380.80859375 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625* proc.memory.percent‚17.403101921081543 network.recv‚ 2.71075e+06IÇx2•!- _wandbruntime‚7696  _runtime‚7696GßArÇ–!Á trainer/global_step‚7949 % _timestamp‚1.7884242253123965e+09 _runtime‚7696.062365772 _step‚650 & lr-AdamW/pg1‚0.0002541528059271605 & lr-AdamW/pg2‚0.0002541528059271605 Ú‹Ì—!Æ & lr-AdamW/pg1‚0.0002541528059271605 & lr-AdamW/pg2‚0.0002541528059271605 trainer/global_step‚7949 % _timestamp‚1.7884242253123965e+09 _runtime‚7696.062365772 _step‚650ŠË3I2˜!- _wandbruntime‚7696  _runtime‚7696U_f(ì'™!æO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146524011 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329444355378176 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206816895 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53486275 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312691199007424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352692828833280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206791225 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139208187 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146524013 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14716823756480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 $ _timestamp‚1.788424225313154e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206791225 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 262530048 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329444355378176 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329444601584640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297974358211584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14759434486784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14759662690304 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146524013 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146524011 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314684938894336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 286470144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14759662690304 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚337676444949504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15016646413824 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _runtime‚7696.063264376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314684920891392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53486955 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312690958318916 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073%Ѩ?¾š!ëO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329444355378176 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329444601584640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297974358211584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14759434486784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14759662690304 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146524013 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146524011 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314684938894336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 286470144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14759662690304 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚337676444949504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15016646413824 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _runtime‚7696.063264376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314684920891392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53486955 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312690958318916 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146524011 R>DeviceStatsMonitor.{ L3on_train_batch_start/active_bytes.all.freed‚329444355378176 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206816895 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53486275 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312691199007424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352692828833280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206791225 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139208187 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146524013 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14716823756480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 $ _timestamp‚1.788424225313154e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206791225 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 262530048 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299‹¿DÐL2›!- _wandbruntime‚7696  _runtime‚7696¨·ü ÷&œ!ñM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85732267 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206816776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚337717638585856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312729333865088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146541732 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329484531016704 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298010660669440 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14718656156288 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146541730 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298010677708800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146541732 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15018256798720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53493270 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 _step‚652 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 353358506 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139224906 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206816776 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312729093176372 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚337717614645760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14761500398080 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 _runtime‚7696.944778037 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314723258835456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 trainer/global_step‚7949 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329484777236480 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚314723276838400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352736162095616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 353384204 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53492639 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314723276838400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14761272181248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14718432506932 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329484777236480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353384204 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561ËX&ªü&!öM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 _runtime‚7696.944778037 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314723258835456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 trainer/global_step‚7949 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329484777236480 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚314723276838400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352736162095616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 353384204 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53492639 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314723276838400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14761272181248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14718432506932 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329484777236480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353384204 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85732267 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206816776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚337717638585856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312729333865088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146541732 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329484531016704 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298010660669440 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14718656156288 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146541730 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298010677708800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146541732 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15018256798720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53493270 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 _step‚652 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 353358506 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139224906 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206816776 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312729093176372 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚337717614645760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14761500398080Œ#Ý{“2ž!- _wandbruntime‚7696  _runtime‚7696—Û~7]Ÿ!× /loss/FBt1/detection_loss‚3.9625113109747567 /loss/FBt3/detection_loss‚2.2139963433146477 0loss/FBt3/assignment_loss‚0.2126411874196492 /loss/FBt4/assignment_loss‚0.198773012496531 /loss/FBt4/detection_loss‚1.8946572482585906 %loss/total_loss‚42.88256173881659 epoch‚6 trainer/global_step‚7949 _step‚653 2loss/SRqqt1/assignment_loss‚1.9746968956329884 1loss/SRqqt3/assignment_loss‚1.568140326761732 1loss/SRqqt3/detection_loss‚1.8795459857758354 0loss/FRt2/assignment_loss‚1.5893555782058022 0loss/FRt3/assignment_loss‚3.0577173843270256 1loss/FBt1/assignment_loss‚0.40005164717634517 0loss/SRqqt4/detection_loss‚2.089363762310573 0loss/FRt4/assignment_loss‚2.7934009902405017 0loss/SRqqt1/detection_loss‚2.378568925918677 % _timestamp‚1.7884242261960492e+09 2loss/SRqqt2/assignment_loss‚0.9399777343375729 0loss/FBt2/assignment_loss‚0.0547038321138206 /loss/FRt1/detection_loss‚2.9542645565234125 /loss/FRt3/detection_loss‚2.6223871388605664 /loss/FRt4/detection_loss‚2.5116059924627407 _runtime‚7696.945868342 /loss/FRt2/detection_loss‚1.8517157784917138 1loss/SRqqt2/detection_loss‚0.8897377711224865 0loss/FBt2/detection_loss‚0.48900760466379944 0loss/FRt1/assignment_loss‚2.9279790706932545 2loss/SRqqt4/assignment_loss‚1.4277616607335706ÁSVb !Ü 0loss/FRt4/assignment_loss‚2.7934009902405017 0loss/SRqqt1/detection_loss‚2.378568925918677 % _timestamp‚1.7884242261960492e+09 2loss/SRqqt2/assignment_loss‚0.9399777343375729 0loss/FBt2/assignment_loss‚0.0547038321138206 /loss/FRt1/detection_loss‚2.9542645565234125 /loss/FRt3/detection_loss‚2.6223871388605664 /loss/FRt4/detection_loss‚2.5116059924627407 _runtime‚7696.945868342 /loss/FRt2/detection_loss‚1.8517157784917138 1loss/SRqqt2/detection_loss‚0.8897377711224865 0loss/FBt2/detection_loss‚0.48900760466379944 0loss/FRt1/assignment_loss‚2.9279790706932545 2loss/SRqqt4/assignment_loss‚1.4277616607335706 /loss/FBt1/detection_loss‚3.9625113109747567 /loss/FBt3/detection_loss‚2.2139963433146477 0loss/FBt3/assignment_loss‚0.2126411874196492 /loss/FBt4/assignment_loss‚0.198773012496531 /loss/FBt4/detection_loss‚1.8946572482585906 %loss/total_loss‚42.88256173881659 epoch‚6 trainer/global_step‚7949 _step‚653 2loss/SRqqt1/assignment_loss‚1.9746968956329884 1loss/SRqqt3/assignment_loss‚1.568140326761732 1loss/SRqqt3/detection_loss‚1.8795459857758354 0loss/FRt2/assignment_loss‚1.5893555782058022 0loss/FRt3/assignment_loss‚3.0577173843270256 1loss/FBt1/assignment_loss‚0.40005164717634517 0loss/SRqqt4/detection_loss‚2.089363762310573’Å0›¡!‚(: ¯àäԙמx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.259' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.251800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&­^¢!‚(:Ó ¯àäÔùÚËz disk.rbd1.out‚ 3.74267578125$ disk.nvme0n1.out‚34783.337890625 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda2.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 334.609375* proc.memory.percent‚17.403101921081543 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚5.3125! disk.rbd2.out‚230.00048828125 disk.nvme0c0n1.in‚5.3125$ memory_percent‚97.95770645141602 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd4.out‚0 cpu‚25.284312519883485 proc.cpu.threads‚17 network.sent‚ 1.9571434e+07 disk.dm-0.in‚5.3125 disk.dm-1.out‚381 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 381.0390625 disk.sda3.in‚7.3125 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625& disk.nvme0c0n1.out‚34783.337890625 network.recv‚ 2.715393e+06$ disk./.usageGB‚1097.357681274414 disk.rbd3.in‚0! disk.dm-0.out‚34783.337890625 disk.dm-1.in‚7.3125# proc.memory.rssMB‚ 2851.32421875) disk./.usagePercent‚73.64459473474834 disk.sda.in‚ 7.36328125 disk.sda3.out‚381 disk.drbd1777.out‚0égwš£!‚(: ¾àäÔ¢Žx gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.46' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.491999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£¥ÙHw¤!‚(:ì ¾àäÔà¦z network.sent‚ 1.9679667e+07 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 381.63671875 disk.rbd1.out‚ 3.74267578125# proc.memory.rssMB‚ 2851.32421875 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚34783.79150390625$ memory_percent‚97.95770645141602! disk.rbd2.out‚230.00048828125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚34783.79150390625 disk.sda2.out‚ 0.0390625% disk./.usageGB‚1097.3577308654785 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 381.67578125* proc.memory.percent‚17.403101921081543 disk.rbd4.in‚0 disk.dm-0.in‚5.3125# disk.dm-0.out‚34783.79150390625 disk.dm-1.in‚7.3125 proc.cpu.threads‚17 network.recv‚ 2.722216e+06 disk.nvme0c0n1.in‚5.3125) disk./.usagePercent‚73.64459806284628 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda2.in‚0 disk.dm-1.out‚ 381.63671875 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125& proc.memory.availableMB‚ 334.609375 cpu‚25.28386107186042¼<€¬›¥!‚(: ÍàäÔ¶‚°x gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.218' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.843600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 LJ(·¦!‚(:å ÍàäÔ–£Õz( disk.nvme0c0n1.out‚34784.45068359375 disk.rbd3.in‚0 network.sent‚ 1.9685992e+07& disk.nvme0n1.out‚34784.45068359375 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda.out‚ 382.1015625 disk.dm-0.in‚5.3125 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125$ memory_percent‚97.95777797698975 proc.cpu.threads‚17# disk.7ܽ‹¹dm-0.out‚34784.45068359375 disk.rbd4.out‚0 disk.sda1.out‚0 network.recv‚ 2.726933e+06 disk.rbd1.out‚ 3.82861328125 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.sda3.out‚382.0625! disk.rbd2.out‚230.14111328125( proc.memory.availableMB‚ 334.59765625# proc.memory.rssMB‚ 2851.32421875 disk.dm-1.out‚382.0625) disk./.usagePercent‚73.64460599907984 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 cpu‚25.28340693339484% disk./.usageGB‚1097.3578491210938 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.sda3.in‚7.3125* proc.memory.percent‚17.403101921081543A,`2§!- _wandbruntime‚7742  _runtime‚7742v©3ùǨ!Á % _timestamp‚1.7884242716829195e+09 _runtime‚7742.432996679 _step‚654 & lr-AdamW/pg1‚0.0002487799769626808 & lr-AdamW/pg2‚0.0002487799769626808 trainer/global_step‚7999'°áaÌ©!Æ & lr-AdamW/pg1‚0.0002487799769626808 & lr-AdamW/pg2‚0.0002487799769626808 trainer/global_step‚7999 % _timestamp‚1.7884242716829195e+09 _runtime‚7742.432996679 _step‚654ŽÃØ,ó2ª!- _wandbruntime‚7742  _runtime‚7742(†¸ßì'«!æO _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14851561827328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354844711617024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14808457327688 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331453151718400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14851333623808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208072127 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86239664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355507760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147409963 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208097797 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316601836097536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147409961 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15109781854720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚7742.433962048 _step‚655 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314597714782924 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147409961 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314597955471432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331453151718400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14851561827328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316601818094592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚299789481104384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86239662 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 208072127 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14851333623808 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140050138 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281404416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14808233678540 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316601818094592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331453397924864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354845016961536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561\ªœQñ'¬!ëO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314597955471432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331453151718400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14851561827328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316601818094592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚299789481104384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86239662 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 208072127 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14851333623808 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140050138 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281404416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14808233678540 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316601818094592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331453397924864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354845016961536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14851561827328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354844711617024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14808457327688 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331453151718400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14851333623808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208072127 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86239664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355507760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147409963 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208097797 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316601836097536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147409961 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15109781854720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚7742.433962048 _step‚655 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314597714782924 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147409961 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0è&û2­!- _wandbruntime‚7743  _runtime‚7743–Áy÷&®!ñM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14853171318272 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15111646166016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚299825783562240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339776123398656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314635849640380 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14853399535104 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147427682 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 355525358 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884242725500436e+09 _runtime‚7743.300742007 _step‚656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208123374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86250019 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 trainer/global_step‚7999 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14853171318272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14810289727496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53816555 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316640174041600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53815936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140066574 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15111923362816 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354887745624576 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355551056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339776099458560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 355551056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331493327356928 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14853399535104 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140065953 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316640156038656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355525358 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316640174041600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354888046761472 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208097678 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316640156038656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14810066078140 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355551056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339776099458560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 355551056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331493327356928 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14853399535104 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140065953 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316640156038656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355525358 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316640174041600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354888046761472 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208097678 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316640156038656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14810066078140 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 trainer/global_step‚7999 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14853171318272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14810289727496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53816555 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316640174041600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53815936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140066574 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15111923362816 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354887745624576:Õ2°!- _wandbruntime‚7743  _runtime‚7743=éá^±!Ø epoch‚6 /loss/FRt4/assignment_loss‚3.268904831520347 2loss/SRqqt2/assignment_loss‚0.8556910923184154 2loss/FBt1/assignment_loss‚0.029273190908133982 1loss/FBt3/assignment_loss‚0.14858911116607487 /loss/FBt1/detection_loss‚3.2914960265159605 0loss/FRt3/assignment_loss‚2.7017758848766484 0loss/FBt2/assignment_loss‚0.1292659710267412 /loss/FRt1/detection_loss‚2.4289377205901674 1loss/FBt4/assignment_loss‚0.17683985644751893 .loss/FBt3/detection_loss‚2.562218278646469 /loss/FBt4/detection_loss‚2.6431321586881364 _step‚657 0loss/SRqqt1/detection_loss‚2.297604406873385 1loss/SRqqt2/detection_loss‚1.0123227553826475 0loss/FBt2/detection_loss‚0.44576571793127134 trainer/global_step‚7999 0loss/FRt2/assignment_loss‚1.2027069055117094 .loss/FRt3/detection_loss‚2.429283525173863 /loss/FRt4/detection_loss‚2.1048417217378215 0loss/FRt1/assignment_loss‚3.8473063508669534 2loss/SRqqt4/assignment_loss‚1.1088196196489863 -loss/FRt2/detection_loss‚1.95863601192832 1loss/SRqqt3/detection_loss‚1.7626485121354722 1loss/SRqqt4/detection_loss‚1.9097214851114486 1loss/SRqqt1/assignment_loss‚1.750206418749359 2loss/SRqqt3/assignment_loss‚1.4757858700396722 &loss/total_loss‚41.541773423795526 % _timestamp‚1.7884242725520976e+09 _runtime‚7743.301960964[Ï…ùc²!Ý .loss/FBt3/detection_loss‚2.562218278646469 /loss/FBt4/detection_loss‚2.6431321586881364 _step‚657 0loss/SRqqt1/detection_loss‚2.297604406873385 1loss/SRqqt2/detection_loss‚1.0123227553826475 0loss/FBt2/detection_loss‚0.44576571793127134 trainer/global_step‚7999 0loss/FRt2/assignment_loss‚1.2027069055117094 .loss/FRt3/detection_loss‚2.429283525173863 /loss/FRt4/detection_loss‚2.1048417217378215 0loss/FRt1/assignment_loss‚3.8473063508669534 2loss/SRqqt4/assignment_loss‚1.1088196196489863 -loss/FRt2/detection_loss‚1.95863601192832 1loss/SRqqt3/detection_loss‚1.7626485121354722 1loss/SRqqt4/detection_loss‚1.9097214851114486 1loss/SRqqt1/assignment_loss‚1.750206418749359 2loss/SRqqt3/assignment_loss‚1.4757858700396722 &loss/total_loss‚41.541773423795526 % _timestamp‚1.7884242725520976e+09 _runtime‚7743.301960964 epoch‚6 /loss/FRt4/assignment_loss‚3.268904831520347 2loss/SRqqt2/assignment_loss‚0.8556910923184154 2loss/FBt1/assignment_loss‚0.029273190908133982 1loss/FBt3/assignment_loss‚0.14858911116607487 /loss/FBt1/detection_loss‚3.2914960265159605 0loss/FRt3/assignment_loss‚2.7017758848766484 0loss/FBt2/assignment_loss‚0.1292659710267412 /loss/FRt1/detection_loss‚2.4289377205901674 1loss/FBt4/assignment_loss‚0.17683985644751893‘v..³!‚(:… ÜàäÔÝÁÃx gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.903' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7806 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šcòsp´!‚(:å ÜàäÔÓµ¤{ disk.rbd3.in‚0 disk.rbd4.in‚0 disk.sda2.out‚ 0.0390625% disk./.usageGB‚1097.3577880859375 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.dm-1.in‚7.3125 disk.rbd3.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2851.32421875 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚5.3125# disk.dm-0.out‚34784.86474609375# memory_percent‚97.9581356048584* proc.memory.percent‚17.403101921081543 network.recv‚ 2.730462e+06( disk./.usagePercent‚73.6446019029593( disk.nvme0c0n1.out‚34784.86474609375 disk.rbd2.in‚0! disk.rbd2.out‚231.71533203125 proc.cpu.threads‚17 disk.sda.out‚382.9375 disk.dm-1.out‚ 382.8984375' proc.memory.availableMB‚ 334.5390625 cpu‚25.28295280355044 network.sent‚ 1.9691283e+07 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.sda3.out‚ 382.8984375 disk.rbd1.out‚ 3.82861328125 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚34784.86474609375 disk.sda2.in‚0 disk.rbd0.out‚0ή¿^µ!‚(:… ëàäÔéÂx gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.896' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ÿxl¶!‚(:á ëàäÔê‚Ï| disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2851.32421875 cpu‚25.282495714749707 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda3.out‚383.25* proc.memory.percent‚17.403101921081543 proc.cpu.threads‚17 disk.nvme0n1.in‚5.3125! disk.rbd2.out‚231.79345703125 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.sda3.in‚7.3125( proc.memory.availableMB‚ 334.79296875 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚34787.28076171875 disk.sda2.in‚0$ memory_percent‚97.95658588409424 network.sent‚ 1.9799842e+07 disk.dm-0.in‚5.3125# disk.dm-0.out‚34787.28076171875 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 3.89111328125) disk./.usagePercent‚73.63322006399451$ disk./.usageGB‚1097.188190460205& disk.nvme0n1.out‚34787.28076171875 disk.dm-1.in‚7.3125 disk.rbd3.out‚0 disk.sda.out‚ 383.2890625 network.recv‚ 2.737759e+06 disk.dm-1.out‚383.25 disk.sda1.out‚0¸ë0'Ž·!‚(:ƒ úàäÔÅžx gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚103.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.724 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý?ßßz¸!‚(:ï úàäÔÓÌÌ{ disk.dm-1.out‚ 383.73828125 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda3.out‚ 383.73828125! disk.rbd2.out‚231.79345703125 cpu‚25.282024111459886 network.sent‚ 1.9805803e+07 network.recv‚ 2.741888e+06 disk.sda2.in‚0( proc.memory.availableMB‚ 334.78515625 disk.sda.in‚ 7.36328125) disk./.usagePercent‚73.63311023676232 disk.rbd4.in‚0# disk.dm-0.out‚34787.75732421875 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625( disk.nvme0c0n1.out‚34787.75732421875 disk.sda1.out‚0 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚34787.75732421875 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2851.32421875% disk./.usageGB‚1097.1865539550781 disk.sda.out‚ 383.77734375 disk.sda3.in‚7.3125 disk.rbd1.out‚ 3.89111328125 disk.dm-1.in‚7.3125$ memory_percent‚97.95663356781006 disk.rbd3.in‚0 disk.dm-0.in‚5.3125* proc.memory.percent‚17.403101921081543 proc.cpu.threads‚17ûs2¹!- _wandbruntime‚7789  _runtime‚7789_㉷ɺ!à ' lr-AdamW/pg2‚0.00024343890362813465 trainer/global_step‚8049 % _timestamp‚1.7884243183685718e+09 _runtime‚7789.118523824 _step‚658 ' lr-AdamW/pg1‚0.00024343890362813465ð%qaλ!È ' lr-AdamW/pg1‚0.00024343890362813465 ' lr-AdamW/pg2‚0.00024343890362813465 trainer/global_step‚8049 % _timestamp‚1.7884243183685718e+09 _runtime‚7789.118523824 _step‚658’gê¿À2¼!- _wandbruntime‚7789  _runtime‚7789c¼rpí'½!çO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333462194265088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚341793414575104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 % _timestamp‚1.7884243183693616e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15203271705088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14899867249748 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148295911 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14900090898896 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140887083 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚318518733300736 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148295913 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318518715297792 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148295911 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148295913 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 357648940 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209378699 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚316504471246932 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333461948058624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209353029 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86757412˜k‹Ã¹¾!ìO UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148295913 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318518715297792 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148295911 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148295913 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 357648940 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209378699 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚316504471246932 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333461948058624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209353029 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86757412 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333462194265088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚341793414575104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 % _timestamp‚1.7884243183693616e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15203271705088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14899867249748 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148295911 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14900090898896 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140887083 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚318518733300736“œz‡2¿!- _wandbruntime‚7790  _runtime‚7790¤^0oö&À!ðM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚318557071244800 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333502369916928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86767769 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54136152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 trainer/global_step‚8049 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚318557053241856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148313632 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚318557053241856 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209404276 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚318557071244800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357717908 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209378580 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148313630 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14901923298704 _step‚660 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148313630 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14945298672128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚301640923494400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚357039737361920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140903919 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357717908 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14945070455296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚357040040595968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚316542606104388 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14945070455296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140904557 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚316542846793104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚301640906455040 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15205153090560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209404276 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14945298672128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 $ _timestamp‚1.788424319261941e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333502369916928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86767769 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54136152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 trainer/global_step‚8049 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚318557053241856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148313632 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚318557053241856 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209404276 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚318557071244800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357717908 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209378580 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148313630 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14901923298704 _step‚660 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148313630 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14945298672128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚301640923494400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚357039737361920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140903919 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357717908 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14945070455296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚357040040595968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚316542606104388 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14945070455296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140904557 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚316542846793104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚301640906455040 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15205153090560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992”ï—Ž2Â!- _wandbruntime‚7790  _runtime‚7790j)gWÃ!Ñ 0loss/FRt3/assignment_loss‚3.3091882583571643 /loss/FRt1/detection_loss‚2.5298197915156684 1loss/SRqqt3/detection_loss‚2.3603123550007985 /loss/FBt1/detection_loss‚2.0739926934242248 .loss/FBt4/detection_loss‚2.049843152364095 &loss/total_loss‚41.016510842620065 % _timestamp‚1.7884243192636983e+09 .loss/FRt4/detection_loss‚2.146343714563671 _step‚661 /loss/FRt1/assignment_loss‚3.997820631051675 0loss/SRqqt1/assignment_loss‚1.57011538464576 2loss/SRqqt3/assignment_loss‚1.5660856987644987 -loss/FRt3/detection_loss‚2.58496187809037 trainer/global_step‚8049 0loss/FRt2/assignment_loss‚1.3607394262856127 1loss/FBt4/assignment_loss‚0.06119673159749558 1loss/SRqqt4/detection_loss‚1.6941672455933359 /loss/FRt4/assignment_loss‚3.181130726086466 2loss/SRqqt2/assignment_loss‚0.9320616576505799 2loss/SRqqt4/assignment_loss‚1.4608065025122077 1loss/FBt2/assignment_loss‚0.09903119544915723 .loss/FRt2/detection_loss‚1.680563434958458 /loss/FBt2/detection_loss‚0.5180073279324312 /loss/FBt3/detection_loss‚2.3273210003972054 1loss/FBt3/assignment_loss‚0.10629681066347985 0loss/SRqqt1/detection_loss‚2.360959365963936 epoch‚6 .loss/FBt1/assignment_loss‚0.17572142494937 1loss/SRqqt2/detection_loss‚0.8700244348023979 _runtime‚7790.013495619’+q\Ä!Ö /loss/FRt4/assignment_loss‚3.181130726086466 2loss/SRqqt2/assignment_loss‚0.9320616576505799 2loss/SRqqt4/assignment_loss‚1.4608065025122077 1loss/FBt2/assignment_loss‚0.09903119544915723 .loss/FRt2/detection_loss‚1.680563434958458 /loss/FBt2/detection_loss‚0.5180073279324312 /loss/FBt3/detection_loss‚2.3273210003972054 1loss/FBt3/assignment_loss‚0.10629681066347985 0loss/SRqqt1/detection_loss‚2.360959365963936 epoch‚6 .loss/FBt1/assignment_loss‚0.17572142494937 1loss/SRqqt2/detection_loss‚0.8700244348023979 _runtime‚7790.013495619 0loss/FRt3/assignment_loss‚3.3091882583571643 /loss/FRt1/detection_loss‚2.5298197915156684 1loss/SRqqt3/detection_loss‚2.3603123550007985 /loss/FBt1/detection_loss‚2.0739926934242248 .loss/FBt4/detection_loss‚2.049843152364095 &loss/total_loss‚41.016510842620065 % _timestamp‚1.7884243192636983e+09 .loss/FRt4/detection_loss‚2.146343714563671 _step‚661 /loss/FRt1/assignment_loss‚3.997820631051675 0loss/SRqqt1/assignment_loss‚1.57011538464576 2loss/SRqqt3/assignment_loss‚1.5660856987644987 -loss/FRt3/detection_loss‚2.58496187809037 trainer/global_step‚8049 0loss/FRt2/assignment_loss‚1.3607394262856127 1loss/FBt4/assignment_loss‚0.06119673159749558 1loss/SRqqt4/detection_loss‚1.6941672455933359•“cÅ!‚(:„ ‰áäÔÕÛ©x gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.235' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.647 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î6óMoÆ!‚(:ä ‰áäÔ—ÆÔ{ disk.dm-1.out‚384.1875 disk.rbd4.out‚0( disk.nvme0c0n1.out‚34921.39404296875 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda1.out‚0# disk.dm-0.out‚34921.39404296875$ memory_percent‚97.95682430267334 proc.cpu.threads‚17) disk./.usagePercent‚73.64183906964979 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125& disk.nvme0n1.out‚34921.39404296875 disk.rbd4.in‚0# proc.memory.rssMB‚ 2851.32421875 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda.out‚ 384.2265625 disk.sda3.in‚7.3125 disk.nvme0n1.in‚5.3125 disk.dm-1.in‚7.3125 disk.rbd2.out‚231.9580078125 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125( proc.memory.availableMB‚ 334.75390625 cpu‚25.28161096543484 network.sent‚ 1.9812194e+07 network.recv‚ 2.746935e+06% disk./.usageGB‚1097.3166198730469 disk.rbd3.in‚0 disk.sda3.out‚384.1875 disk.dm-0.in‚5.3125* proc.memory.percent‚17.403101921081543 disk.rbd1.out‚ 3.92333984375 µU\šÇ!‚(: ˜áäÔ϶¬x gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.46' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.491999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú‰ômxÈ!‚(:í ˜áäÔüœî{ disk.sda.in‚ 7.36328125 disk.sda.out‚ 384.89453125& disk.nvme0n1.out‚35054.87451171875 network.recv‚ 2.752933e+06 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.sda3.in‚7.3125 disk.sda3.out‚ 384.85546875 disk.rbd1.in‚0 network.sent‚ 1.991988e+07 disk.rbd2.out‚231.9580078125( disk.nvme0c0n1.out‚35054.87451171875 disk.rbd1.out‚ 3.92333984375 cpu‚25.281166530302308 disk.sda2.out‚ 0.0390625 disk.rbd4.out‚0 disk.nvme0n1.in‚5.3125 disk.drbd1777.out‚0 disk.sda1.out‚0# disk.dm-0.out‚35054.87451171875 disk.dm-1.out‚ 384.85546875 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.65058249496668 disk.rbd3.in‚0$ memory_percent‚97.95458316802979( proc.memory.availableMB‚ 335.12109375# proc.memory.rssMB‚ 2851.32421875% disk./.usageGB‚1097.4469032287598 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.dm-0.in‚5.3125* proc.memory.percent‚17.403101921081543 disk.sda2.in‚0 disk.rbd3.out‚0#Wm¬É!‚(:„ §áäÔû®x gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UÆPŒlÊ!‚(:á §áäÔã–¿{ disk.nvme0c0n1.in‚5.3125' disk.nvme0c0n1.out‚35055.7548828125 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda3.in‚7.3125 network.recv‚ 2.757473e+06% disk./.usageGB‚1097.4468841552734 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125" disk.dm-0.out‚35055.7548828125 disk.rbd2.in‚0 disk.rbd2.out‚233.5478515625 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 385.265625% disk.nvme0n1.out‚35055.7548828125# proc.memory.rssMB‚ 2851.32421875 proc.cpu.threads‚17' disk./.usagePercent‚73.650581214929 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.drbd1777.out‚0 disk.nvme0n1.in‚5.3125* proc.memory.percent‚17.403101921081543 disk.rbd4.in‚0$ memory_percent‚97.95835018157959 disk.rbd3.in‚0 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625( proc.memory.availableMB‚ 334.50390625 cpu‚25.28072760209048 disk.rbd3.out‚0 disk.sda.out‚ 385.3046875 network.sent‚ 1.9925897e+07 disk.dm-1.out‚ 385.265625þRö!2Ë!- _wandbruntime‚7836  _runtime‚7836yLt¢ÇÌ!Á _runtime‚7836.327935488 _step‚662 ' lr-AdamW/pg1‚0.00023813069899519956 ' lr-AdamW/pg2‚0.00023813069899519956 trainer/global_step‚8099 # _timestamp‚1.78842436557794e+09óñ~ÌÍ!Æ ' lr-AdamW/pg2‚0.00023813069899519956 trainer/global_step‚8099 # _timestamp‚1.78842436557794e+09 _runtime‚7836.327935488 _step‚662 ' lr-AdamW/pg1‚0.00023813069899519956–bË]þ2Î!- _wandbruntime‚7836  _runtime‚7836†¢÷?Ï!çO H?DeviceStatsMonitor.on_train_batch_start/inactive_spliádÚÌ®'t.all.peak‚1261 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335470744398848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54451931 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚359148808048640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚318411468399448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 trainer/global_step‚8099 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210633931 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335470990605312 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87275164 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚303419743929344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14991500820956 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320435630503936 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141727095 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14991724470104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335470990605312 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15035360101376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141726427 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318411227710940 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 359815792 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15035131897856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚666 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149181861 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54451265 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚303419726889984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15035360101376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335470744398848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210659601 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15035131897856 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149181863 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚320435612500992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256¢`2±ò'Ð!ìO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚320435612500992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335470744398848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54451931 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚359148808048640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚318411468399448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 trainer/global_step‚8099 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210633931 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335470990605312 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87275164 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚303419743929344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14991500820956 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320435630503936 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141727095 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14991724470104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335470990605312 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15035360101376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141726427 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318411227710940 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 359815792 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15035131897856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚666 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149181861 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54451265 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚303419726889984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15035360101376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335470744398848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210659601 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15035131897856 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149181863 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168—*°Ï2Ñ!- _wandbruntime‚7837  _runtime‚7837Êiÿ÷&Ò!ñM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 307428352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15298766912000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 359859062 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15299050400256 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15036969592320 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335511166257152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚303456029347840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 359859062 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149199582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚8099 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149199580 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15036969592320 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320473950445056 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15037197809152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14993333220556 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87285519 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15037197809152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210659482 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149199582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335511166257152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149199580 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 283488256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318449362568396 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335510920037376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141743537 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54457395 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359192143424512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359191835996160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14993556869912 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚664 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 359884760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚303456046387200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210659482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210685178 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87285517 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚320473968448000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141742912ËN¢® Ó!öM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 359884760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚303456046387200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210659482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210685178 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87285517 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚320473968448000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141742912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 307428352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15298766912000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 359859062 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15299050400256 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15036969592320 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335511166257152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚303456029347840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 359859062 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149199582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚8099 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149199580 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15036969592320 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320473950445056 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15037197809152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14993333220556 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87285519 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15037197809152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210659482 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149199582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335511166257152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149199580 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 283488256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318449362568396 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335510920037376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141743537 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54457395 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359192143424512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359191835996160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14993556869912 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚664 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896˜¶)¤2Ô!- _wandbruntime‚7837  _runtime‚7837BË x[Õ!Õ trainer/global_step‚8099 /loss/FRt1/assignment_loss‚3.720842612981796 _runtime‚7837.199387792 1loss/SRqqt1/detection_loss‚2.1765266697149017 0loss/SRqqt3/detection_loss‚1.775065819804485 1loss/SRqqt4/detection_loss‚1.9939063144442828 % _timestamp‚1.7884243664495697e+09 0loss/FRt2/assignment_loss‚1.5636902976036071 0loss/FBt3/assignment_loss‚0.4044117562007159 _step‚665 1loss/SRqqt3/assignment_loss‚1.507677371542041 0loss/FRt4/assignment_loss‚2.9471377590123344 2loss/SRqqt1/assignment_loss‚1.8481041307787638 2loss/SRqqt2/assignment_loss‚0.8606415831018239 0loss/FBt1/assignment_loss‚0.9243518728762865 0loss/FBt2/assignment_loss‚0.1308810072759393 .loss/FBt1/detection_loss‚2.804702954632895 /loss/FBt3/detection_loss‚2.4128285571932793 epoch‚6 2loss/SRqqt4/assignment_loss‚1.2431171959994927 /loss/FRt2/detection_loss‚1.9677825200557708 /loss/FRt3/detection_loss‚2.7566162271159036 /loss/FBt4/detection_loss‚1.9794147339734165 &loss/total_loss‚41.775355444171495 0loss/FRt3/assignment_loss‚2.5447934644562857 1loss/FBt4/assignment_loss‚0.15874661899976095 .loss/FRt1/detection_loss‚2.409538197517395 /loss/FRt4/detection_loss‚2.0028290739495906 0loss/SRqqt2/detection_loss‚0.883947805443313 /loss/FBt2/detection_loss‚0.7578008994974148ÕÍ`Ö!Ú /loss/FBt3/detection_loss‚2.4128285571932793 epoch‚6 2loss/SRqqt4/assignment_loss‚1.2431171959994927 /loss/FRt2/detection_loss‚1.9677825200557708 /loss/FRt3/detection_loss‚2.7566162271159036 /loss/FBt4/detection_loss‚1.9794147339734165 &loss/total_loss‚41.775355444171495 0loss/FRt3/assignment_loss‚2.5447934644562857 1loss/FBt4/assignment_loss‚0.15874661899976095 .loss/FRt1/detection_loss‚2.409538197517395 /loss/FRt4/detection_loss‚2.0028290739495906 0loss/SRqqt2/detection_loss‚0.883947805443313 /loss/FBt2/detection_loss‚0.7578008994974148 trainer/global_step‚8099 /loss/FRt1/assignment_loss‚3.720842612981796 _runtime‚7837.199387792 1loss/SRqqt1/detection_loss‚2.1765266697149017 0loss/SRqqt3/detection_loss‚1.775065819804485 1loss/SRqqt4/detection_loss‚1.9939063144442828 % _timestamp‚1.7884243664495697e+09 0loss/FRt2/assignment_loss‚1.5636902976036071 0loss/FBt3/assignment_loss‚0.4044117562007159 _step‚665 1loss/SRqqt3/assignment_loss‚1.507677371542041 0loss/FRt4/assignment_loss‚2.9471377590123344 2loss/SRqqt1/assignment_loss‚1.8481041307787638 2loss/SRqqt2/assignment_loss‚0.8606415831018239 0loss/FBt1/assignment_loss‚0.9243518728762865 0loss/FBt2/assignment_loss‚0.1308810072759393 .loss/FBt1/detection_loss‚2.804702954632895™o`mòš×!‚(: ¶áäԮݟx gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.821' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.964199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u¦ÁSiØ!‚(:Þ ¶áäÔ¥íÊ{ disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd3.out‚0 cpu‚25.280287359192027 network.recv‚ 2.762067e+06" disk.dm-0.out‚35056.1904296875 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd3.in‚0' proc.memory.availableMB‚ 334.6953125 disk.rbd4.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚35056.1904296875* proc.memory.percent‚17.403101921081543 disk.sda.out‚ 385.8203125 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 385.78125 disk.rbd0.in‚0 disk.sda.in‚ 7.36328125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd4.out‚0 disk.sda3.out‚ 385.78125 disk.rbd2.out‚233.5478515625 disk.rbd0.out‚0 network.sent‚ 1.9932e+07) disk./.usagePercent‚73.65058505504203% disk./.usageGB‚1097.4469413757324 disk.sda3.in‚7.3125 disk.rbd1.out‚ 3.986328125 disk.sda2.in‚0$ memory_percent‚97.95718193054199 disk.nvme0c0n1.in‚5.3125' disk.nvme0c0n1.out‚35056.1904296875# proc.memory.rssMB‚ 2851.32421875 proc.cpu.threads‚179ÿ[Ù!‚(:„ ÅáäÔÝà˜x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦t[~pÚ!‚(:å ÅáäÔýûÑz disk.rbd4.out‚0% proc.memory.availableMB‚ 334.40625# proc.memory.rssMB‚ 2851.32421875* proc.memory.percent‚17.403101921081543 disk.drbd1777.out‚0 disk.rbd1.in‚0% disk./.usageGB‚1097.4544296264648 disk.rbd2.in‚0 cpu‚25.279855615926902 disk.sda3.out‚ 386.11328125 disk.nvme0n1.in‚5.3125$ disk.nvme0n1.out‚35057.076171875) disk./.usagePercent‚73.65108759783175 disk.sda1.out‚0! disk.dm-0.out‚35057.076171875$ memory_percent‚97.95894622802734 proc.cpu.threads‚17 network.sent‚ 2.0040906e+07 disk.sda.out‚ 386.15234375& disk.nvme0c0n1.out‚35057.076171875 disk.rbd0.out‚0 network.recv‚ 2.769612e+06 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125 disk.sda2.in‚0 disk.dm-1.out‚ 386.11328125 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd1.out‚ 4.04931640625 disk.dm-0.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚7.3125 disk.rbd2.out‚233.6259765625 disk.sda.in‚ 7.36328125f(ÂäŽÛ!‚(:ƒ ÔáäÔ±äŸx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓäæJlÜ!‚(:á ÔáäÔ›¿…{& disk.nvme0n1.out‚35069.00146484375 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 386.7890625 disk.dm-0.in‚5.3125 disk.dm-1.in‚7.3125# proc.memory.rssMB‚ 2851.32421875 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 4.04931640625 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚35069.00146484375 disk.dm-1.out‚386.75 disk.rbd0.in‚0 disk.rbd4.in‚0* proc.memory.percent‚17.403101921081543 disk.sda3.out‚386.75) disk./.usagePercent‚73.65129035579886 disk.sda3.in‚7.3125 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚233.6259765625 disk.drbd1777.out‚0 network.recv‚ 2.773086e+06% disk./.usageGB‚1097.4574508666992 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚35069.00146484375 network.sent‚ 2.0046117e+07 disk.rbd3.in‚0$ memory_percent‚97.95944690704346( proc.memory.availableMB‚ 334.32421875 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 cpu‚25.279416549032828uãØ2Ý!- _wandbruntime‚7882  _runtime‚7882‘˜ýûÉÞ!à % _timestamp‚1.7884244116550868e+09 _runtime‚7882.405014542 _step‚666 ' lr-AdamW/pg1‚0.00023285646928576501 ' lr-AdamW/pg2‚0.00023285646928576501 trainer/global_step‚8149Ûâ’ïÎß!È _step‚666 ' lr-AdamW/pg1‚0.00023285646928576501 ' lr-AdamW/pg2‚0.00023285646928576501 trainer/global_step‚8149 % _timestamp‚1.7884244116550868e+09 _runtime‚7882.405014542š1‹Ô2à!- _wandbruntime‚7882  _runtime‚7882GïÉ?ì'á!æO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15127259238400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚337479786945536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337479540739072 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211940503 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚320317984174948 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15390757908480 _runtime‚ 7882.40600043 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211940503 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337479540739072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚345910408140800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281404416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150067813 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15127259238400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54777034 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884244116558223e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87792914 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚322352527707136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150067813 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15127031034880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305234849782784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54777676 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337479786945536 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 362008316 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87792912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 trainer/global_step‚8149 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15127031034880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345910384200704 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211914833 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211914833 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142569946 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚322352527707136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150067811 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 361982644 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305234866822144 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15083358041312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚361300860704768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _step‚667 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87792912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 trainer/global_step‚8149 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15127031034880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345910384200704 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211914833 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211914833 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142569946 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚322352527707136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150067811 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 361982644 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305234866822144 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15083358041312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚361300860704768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _step‚667 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337479540739072 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211940503 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚320317984174948 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15390757908480 _runtime‚ 7882.40600043 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211940503 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337479540739072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚345910408140800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281404416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150067813 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15127259238400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464›•÷çŒ2ã!- _wandbruntime‚7883  _runtime‚7883ô–Á÷&ä!ñM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15128868729344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211966080 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337519716377600 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150085532 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚322390865651200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚322390847648256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361344195621888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 _step‚668 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337519962597376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54783890 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15392346879488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8149 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15128868729344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362025914 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 211966080 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15085190441120 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150085530 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362051612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87803269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54783254 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142587159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7883.268503337 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15084966791764 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 % _timestamp‚1.7884244125179653e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150085532 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142586521 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345951577837056 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211940384 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚322390865651200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87803267 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305271169280000 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362051612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87803269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54783254 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142587159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7883.268503337 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15084966791764 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 % _timestamp‚1.7884244125179653e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150085532 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142586521 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345951577837056 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211940384 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚322390865651200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87803267 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305271169280000 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211966080 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337519716377600 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150085532 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚322390865651200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚322390847648256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361344195621888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 _step‚668 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337519962597376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54783890 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15392346879488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8149 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15128868729344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362025914 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 211966080 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15085190441120 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150085530 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ]OD±‘"eviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚307049972675584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15218930171904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚347968869013504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324269406907392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55104085 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚347968892953600 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322224981327464 _runtime‚7928.642496113 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143414749 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364149496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213221405 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15219158375424 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213195735 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88310662 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚307049989714944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚339488583285760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150953763 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339488583285760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324269424910336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150953763 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88310664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚363452874420736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚322224740638956 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339488337079296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150953761 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339488583285760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324269424910336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150953763 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88310664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚363452874420736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚322224740638956 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339488337079296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150953761 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚307049972675584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15218930171904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚347968869013504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324269406907392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55104085 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚347968892953600 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322224981327464 _runtime‚7928.642496113 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143414749 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364149496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213221405 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15219158375424 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213195735 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88310662 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚307049989714944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚339488583285760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150953763 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 _runtime‚ 7929.50626279 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324307762854400 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55110240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55109603 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150971480 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚363495904823808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 364192766 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213221286 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143430620 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143431259 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚307086292172800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364218464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _step‚672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15220767866368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88321017 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚8199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324307744851456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324307744851456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15176600362972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213246982 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15486147505152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15485866114048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 % _timestamp‚1.7884244587556422e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚339528512717824 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363496210155008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348010062649856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚307086275133440 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚322262875496412 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 364218464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 364192766 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213221286 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150971482 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15220767866368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348010038709760 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0TÖá2÷!õM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 364192766 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213221286 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143430620 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143431259 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚307086292172800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364218464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _step‚672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15220767866368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88321017 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚8199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324307744851456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324307744851456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15176600362972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213246982 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15486147505152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15485866114048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 % _timestamp‚1.7884244587556422e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚339528512717824 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363496210155008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348010062649856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚307086275133440 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚322262875496412 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 364218464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 364192766 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213221286 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150971482 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15220767866368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348010038709760 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 305331200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚322263116185128 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15176824012328 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150971480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88321019 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15220996083200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 _runtime‚ 7929.50626279 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324307762854400 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55110240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55109603 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150971480 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚363495904823808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 Ô–Õe2ø!- _wandbruntime‚7929  _runtime‚7929Ä®Zù!Ô % _timestamp‚1.7884244587575562e+09 /loss/FRt2/detection_loss‚1.3970037346705795 1loss/SRqqt4/detection_loss‚2.0394423498047725 %loss/total_loss‚40.44479548387662 epoch‚6 /loss/FBt2/detection_loss‚0.6131881235749461 1loss/SRqqt2/assignment_loss‚0.861633729392832 0loss/FRt1/assignment_loss‚2.9488118573238977 /loss/FRt1/detection_loss‚2.5429364278128275 0loss/SRqqt1/detection_loss‚2.183839130240518 1loss/SRqqt3/detection_loss‚1.5887746498232982 .loss/FBt4/detection_loss‚3.222665114062173 0loss/FRt2/assignment_loss‚1.3238026022911071 /loss/FRt4/assignment_loss‚3.501212952714978 /loss/FRt4/detection_loss‚2.4978645735069662 1loss/SRqqt2/detection_loss‚1.3450574219904163 _step‚673 1loss/SRqqt3/assignment_loss‚1.700754092059663 2loss/SRqqt4/assignment_loss‚1.3509789545089006 1loss/FBt1/assignment_loss‚0.37999860743639147 2loss/FBt4/assignment_loss‚0.003858619702181646 -loss/FRt3/detection_loss‚2.24297310743067 /loss/FBt3/detection_loss‚1.8511009487238796 0loss/FBt2/assignment_loss‚0.0673982190126881 1loss/FBt3/assignment_loss‚0.14895359881814907 trainer/global_step‚8199 _runtime‚7929.507384113 0loss/FRt3/assignment_loss‚2.4672840381110155 1loss/SRqqt1/assignment_loss‚1.473415045722111 .loss/FBt1/detection_loss‚2.691847585141659B¦‘_ú!Ù 1loss/SRqqt2/assignment_loss‚0.861633729392832 0loss/FRt1/assignment_loss‚2.9488118573238977 /loss/FRt1/detection_loss‚2.5429364278128275 0loss/SRqqt1/detection_loss‚2.183839130240518 1loss/SRqqt3/detection_loss‚1.5887746498232982 .loss/FBt4/detection_loss‚3.222665114062173 0loss/FRt2/assignment_loss‚1.3238026022911071 /loss/FRt4/assignment_loss‚3.501212952714978 /loss/FRt4/detection_loss‚2.4978645735069662 1loss/SRqqt2/detection_loss‚1.3450574219904163 _step‚673 1loss/SRqqt3/assignment_loss‚1.700754092059663 2loss/SRqqt4/assignment_loss‚1.3509789545089006 1loss/FBt1/assignment_loss‚0.37999860743639147 2loss/FBt4/assignment_loss‚0.003858619702181646 -loss/FRt3/detection_loss‚2.24297310743067 /loss/FBt3/detection_loss‚1.8511009487238796 0loss/FBt2/assignment_loss‚0.0673982190126881 1loss/FBt3/assignment_loss‚0.14895359881814907 trainer/global_step‚8199 _runtime‚7929.507384113 0loss/FRt3/assignment_loss‚2.4672840381110155 1loss/SRqqt1/assignment_loss‚1.473415045722111 .loss/FBt1/detection_loss‚2.691847585141659 % _timestamp‚1.7884244587575562e+09 /loss/FRt2/detection_loss‚1.3970037346705795 1loss/SRqqt4/detection_loss‚2.0394423498047725 %loss/total_loss‚40.44479548387662 epoch‚6 /loss/FBt2/detection_loss‚0.6131881235749461¡ ¶Çû!‚(:… âäÔœ‡±x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö¤¢qü!‚(:æ âäÔ¯Äðz disk.sda1.in‚ 0.05078125 disk.sda1.out‚0# disk.dm-0.out‚35268.56787109375 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625# proc.memory.rssMB‚ 2851.32421875* proc.memory.percent‚17.403101921081543) disk./.usagePercent‚73.68503214881173 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚35268.56787109375 disk.sda.in‚ 7.36328125 network.recv‚ 2.79885e+06 disk.sda2.out‚ 0.0390625 disk.sda.out‚ 388.8828125$ memory_percent‚97.95944690704346 disk.sda3.in‚7.3125 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚35268.56787109375 disk.rbd2.in‚0 network.sent‚ 2.017462e+07% disk./.usageGB‚1097.9602279663086 disk.rbd3.out‚0( proc.memory.availableMB‚ 334.32421875 disk.sda2.in‚0 disk.dm-1.out‚ 388.84375 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 cpu‚25.277625872472314 disk.sda3.out‚ 388.84375 disk.rbd1.out‚ 4.04931640625! disk.rbd2.out‚235.44677734375 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 proc.cpu.threads‚17Ó%”Žý!‚(:ƒ ŸâäÔŸóŽx gpu.0.gpu‚18.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.37' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_I>´hþ!‚(:Ý ŸâäÔëøÌz disk.rbd4.out‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚5.3125 disk.rbd4.in‚0 disk.sda3.out‚389.5# disk.dm-0.out‚35367.57958984375* proc.memory.percent‚17.403101921081543 disk.dm-1.in‚7.3125 disk.rbd2.out‚ 235.615234375 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625( proc.memory.availableMB‚ 334.56640625 network.recv‚ 2.806047e+06) disk./.usagePercent‚73.68496379480008% disk./.usageGB‚1097.9592094421387 disk.rbd3.out‚0 disk.dm-0.in‚5.3125 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2851.32421875 cpu‚25.27716832227507( disk.nvme0c0n1.out‚35367.57958984375 disk.sda1.out‚0 disk.sda.out‚ 389.5390625& disk.nvme0n1.out‚35367.57958984375 disk.dm-1.out‚389.5 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda3.in‚7.3125 disk.rbd1.out‚ 4.04931640625 disk.rbd2.in‚0 network.sent‚ 2.0283151e+07 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚5.3125$ memory_percent‚97.95796871185303¬£ !ÿ!‚(:‚ ®âäÔÒ ‹x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚96.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#ÃOõs€"‚(:è ®âäÔó×±z% disk./.usageGB‚1097.9592514038086 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 390.078125( proc.memory.availableMB‚ 334.26171875 cpu‚25.27671204272028 disk.rbd2.in‚0* proc.memory.percent‚17.403101921081543) disk./.usagePercent‚73.68496661088297 disk.rbd1.out‚ 4.04931640625 disk.dm-1.out‚ 390.0390625 proc.cpu.threads‚17 disk.sda3.out‚ 390.0390625& disk.nvme0n1.out‚35466.93896484375 disk.sda2.in‚0 disk.dm-1.in‚7.3125$ memory_percent‚97.95982837677002# proc.memory.rssMB‚ 2851.32421875 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚5.3125# disk.dm-0.out‚35466.93896484375 network.recv‚ 2.810138e+06 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚5.3125 network.sent‚ 2.0289034e+07 disk.rbd0.out‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚35466.93896484375 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 235.615234375<ñÑH2"- _wandbruntime‚7974  _runtime‚7974ÿ“1[É‚"à _runtime‚7974.760406394 _step‚674 ' lr-AdamW/pg1‚0.00022241432389427936 ' lr-AdamW/pg2‚0.00022241432389427936 trainer/global_step‚8249 % _timestamp‚1.7884245040104504e+09‰Ö©u΃"È ' lr-AdamW/pg2‚0.00022241432389427936 trainer/global_step‚8249 % _timestamp‚1.7884245040104504e+09 _runtime‚7974.760406394 _step‚674 ' lr-AdamW/pg1‚0.00022241432389427936¢ hz—2„"- _wandbruntime‚7974  _runtime‚7974ÀÝOí'…"çO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15311057512448 % _timestamp‚1.7884245040111923e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15577750678528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341497133419520 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15311057512448 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151839711 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15577471371264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55423234 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366316348 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15310829308928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88828412 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚326186304110592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚341497379625984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15266625183728 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15266401534580 _step‚675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168ÿÑ>óý†"ìO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15310829308928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88828412 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚326186304110592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚341497379625984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15266625183728 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15266401534580 _step‚675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15311057512448 % _timestamp‚1.7884245040111923e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15577750678528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341497133419520 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15311057512448 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151839711 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15577471371264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55423234 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366316348 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148£ƒÏ2‡"- _wandbruntime‚7975  _runtime‚7975Ls¿J÷&ˆ"ñM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88838769 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214527884 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55429692 _runtime‚7975.655737884 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214527884 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚324169631960420 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚326224642054656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341537555277824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15579612439040 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚326224660057600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚324169872649136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341537555277824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308901415065600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214502188 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341537309058048 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15312667003392 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144268459 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151857430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 214502188 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55430318 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366385316 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15268233934180 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884245049047942e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144269087 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚365648159901696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308901398026240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15268457583536 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366359618 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151857432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚326224642054656 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151857430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151857432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 trainer/global_step‚82492ö¿Ÿü&‰"öM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214502188 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341537309058048 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15312667003392 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144268459 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151857430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 214502188 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55430318 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366385316 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15268233934180 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884245049047942e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144269087 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚365648159901696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308901398026240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15268457583536 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366359618 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151857432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚326224642054656 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151857430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151857432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 trainer/global_step‚8249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88838769 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214527884 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55429692 _runtime‚7975.655737884 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214527884 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚324169631960420 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚326224642054656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341537555277824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15579612439040 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚326224660057600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚324169872649136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341537555277824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308901415065600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776¤†àϱ2Š"- _wandbruntime‚7975  _runtime‚7975ÐKÛW‹"Ñ 1loss/FBt1/assignment_loss‚0.16902030533153195 /loss/FRt1/detection_loss‚2.6975775028978077 1loss/SRqqt1/detection_loss‚2.2184944788092062 % _timestamp‚1.7884245049071305e+09 1loss/SRqqt1/assignment_loss‚1.212886631880936 0loss/FBt2/assignment_loss‚0.2146745263708626 1loss/FBt3/assignment_loss‚0.11149914709052869 .loss/FRt3/detection_loss‚2.938308329293222 1loss/SRqqt2/detection_loss‚0.8897943335099959 _step‚677 /loss/FRt4/detection_loss‚2.5067958669485075 .loss/FBt1/detection_loss‚2.425230851540199 .loss/FBt3/detection_loss‚2.796878933906555 trainer/global_step‚8249 0loss/SRqqt3/detection_loss‚2.215429901271253 epoch‚6 /loss/FRt1/assignment_loss‚2.472136870452336 /loss/FRt2/detection_loss‚1.9689580576287375 .loss/FBt4/detection_loss‚2.455162247021993 0loss/SRqqt4/detection_loss‚1.997536613182588 /loss/FRt2/assignment_loss‚1.194656511147817 /loss/FRt3/assignment_loss‚2.814760466416677 2loss/SRqqt3/assignment_loss‚1.7097007452032051 _runtime‚7975.657045157 0loss/FRt4/assignment_loss‚3.3789747393492497 2loss/SRqqt2/assignment_loss‚0.9068400393122099 1loss/SRqqt4/assignment_loss‚1.369167384268208 0loss/FBt4/assignment_loss‚0.2866942937317718 0loss/FBt2/detection_loss‚0.44780438048341736 %loss/total_loss‚41.39898315704882Éß5\Œ"Ö .loss/FBt3/detection_loss‚2.796878933906555 trainer/global_step‚8249 0loss/SRqqt3/detection_loss‚2.215429901271253 epoch‚6 /loss/FRt1/assignment_loss‚2.472136870452336 /loss/FRt2/detection_loss‚1.9689580576287375 .loss/FBt4/detection_loss‚2.455162247021993 0loss/SRqqt4/detection_loss‚1.997536613182588 /loss/FRt2/assignment_loss‚1.194656511147817 /loss/FRt3/assignment_loss‚2.814760466416677 2loss/SRqqt3/assignment_loss‚1.7097007452032051 _runtime‚7975.657045157 0loss/FRt4/assignment_loss‚3.3789747393492497 2loss/SRqqt2/assignment_loss‚0.9068400393122099 1loss/SRqqt4/assignment_loss‚1.369167384268208 0loss/FBt4/assignment_loss‚0.2866942937317718 0loss/FBt2/detection_loss‚0.44780438048341736 %loss/total_loss‚41.39898315704882 1loss/FBt1/assignment_loss‚0.16902030533153195 /loss/FRt1/detection_loss‚2.6975775028978077 1loss/SRqqt1/detection_loss‚2.2184944788092062 % _timestamp‚1.7884245049071305e+09 1loss/SRqqt1/assignment_loss‚1.212886631880936 0loss/FBt2/assignment_loss‚0.2146745263708626 1loss/FBt3/assignment_loss‚0.11149914709052869 .loss/FRt3/detection_loss‚2.938308329293222 1loss/SRqqt2/detection_loss‚0.8897943335099959 _step‚677 /loss/FRt4/detection_loss‚2.5067958669485075 .loss/FBt1/detection_loss‚2.425230851540199¥ µ×'›"‚(: ½âäÔšõ”x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.467' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.293400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìM)ŒmŽ"‚(:â ½âäÔïûÃz disk.drbd1777.out‚0% disk./.usageGB‚1097.9592323303223 disk.rbd3.out‚0 disk.rbd1.in‚0" disk.dm-0.out‚35566.1142578125' disk.nvme0c0n1.out‚35566.1142578125( proc.memory.availableMB‚ 334.01171875 network.sent‚ 2.029532e+07% disk.nvme0n1.out‚35566.1142578125 disk.sda2.in‚0 disk.dm-1.out‚ 390.5390625 disk.rbd3.in‚0 disk.sda3.out‚ 390.5390625 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125( disk./.usagePercent‚73.6849653308453 disk.sda.out‚ 390.578125 disk.rbd0.in‚0$ memory_percent‚97.96135425567627 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 4.04931640625 disk.nvme0n1.in‚5.3125 disk.dm-1.in‚7.3125 disk.rbd4.in‚0# proc.memory.rssMB‚ 2851.32421875 cpu‚25.27628635961052 network.recv‚ 2.81497e+06 disk.sda3.in‚7.3125 disk.dm-0.in‚5.3125 disk.rbd2.out‚ 235.814453125* proc.memory.percent‚17.403101921081543 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.rbd0.out‚0 disk.rbd4.out‚0»‰ò"‚(:… ÌâäÔð¯Áx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.149' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾Ïzs"‚(:è ÌâäÔÒ°Þz# proc.memory.rssMB‚ 2851.32421875 disk.rbd3.out‚0& disk.nvme0n1.out‚35567.22509765625 network.recv‚ 2.821282e+06 disk.rbd4.out‚0( disk.nvme0c0n1.out‚35567.22509765625 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0( proc.memory.availableMB‚ 333.98046875 cpu‚25.27583069245102# disk.dm-0.out‚35567.22509765625 disk.rbd2.out‚ 237.388671875 disk.drbd1777.in‚ 1.03515625$ memory_percent‚97.96154499053955 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.rbd1.out‚ 4.04931640625 disk.nvme0n1.in‚5.3125* proc.memory.percent‚17.403101921081543 network.sent‚ 2.0403241e+07 disk.rbd4.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.71867025877323 disk.dm-0.in‚5.3125 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 391.3203125 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.sda.out‚ 391.359375% disk./.usageGB‚1098.4614601135254 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda3.out‚ 391.3203125í Šê‘"‚(:… ÛâäÔæµx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.602' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 C‡év’"‚(:ë ÛâäÔêˆô{ disk.rbd0.out‚0# disk.dm-0.out‚35666.26416015625( disk.nvme0c0n1.out‚35666.26416015625 disk.nvme0c0n1.in‚5.3125 disk.nvme0n1.in‚5.3125 disk.dm-1.in‚7.3125( proc.memory.availableMB‚ 334.23046875 network.recv‚ 2.825633e+06 disk.drbd1777.in‚ 1.03515625 disk.sda2.out‚ 0.0390625 cpu‚25.27543268034141 proc.cpu.threads‚17 network.sent‚ 2.0409365e+07 disk.rbd2.in‚0 disk.rbd2.out‚ 237.466796875 disk.sda3.out‚ 391.66796875# proc.memory.rssMB‚ 2851.32421875 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 391.70703125 disk.sda2.in‚0 disk.dm-1.out‚ 391.66796875* proc.memory.percent‚17.403101921081543 disk.dm-0.in‚5.3125 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚35666.26416015625% disk./.usageGB‚1098.4594268798828 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.rbd1.in‚0# memory_percent‚97.9600191116333) disk./.usagePercent‚73.71853380675748 disk.rbd1.out‚ 4.04931640625GÂ12“"- _wandbruntime‚8021  _runtime‚80211€PÉ”"à % _timestamp‚1.7884245502928617e+09 _runtime‚8021.042770385 _step‚678 ' lr-AdamW/pg1‚0.00021724858433967711 ' lr-AdamW/pg2‚0.00021724858433967711 trainer/global_step‚8299¡LzΕ"È ' lr-AdamW/pg1‚0.00021724858433967711 ' lr-AdamW/pg2‚0.00021724858433967711 trainer/global_step‚8299 % _timestamp‚1.7884245502928617e+09 _runtime‚8021.042770385 _step‚678¦Û iî2–"- _wandbruntime‚8021  _runtime‚8021c3Ýw —"çO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚343506175966208 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145094484 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152725663 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15358258754936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.pHRBveak‚561 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚8299 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215783209 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 215783209 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343505929759744 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89346164 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15671243922944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343506175966208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚352085862579200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343505929759744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310680235500544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152725661 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89346162 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55747685 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215757539 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15402728445952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚328103219316736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 368508872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55748320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328103219316736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367756813740544 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326038253566972 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145093847 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 368483200 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328103201313792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚310680218461184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152725661 _runtime‚8021.043704766 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397÷'ò'˜"ìO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 215783209 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343505929759744 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89346164 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15671243922944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343506175966208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚352085862579200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343505929759744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310680235500544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152725661 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89346162 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55747685 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215757539 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15402728445952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚328103219316736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 368508872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55748320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328103219316736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367756813740544 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326038253566972 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145093847 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 368483200 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328103201313792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚310680218461184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152725661 _runtime‚8021.043704766 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚343506175966208 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145094484 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152725663 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15358258754936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚8299 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215783209§>¥äU2™"- _wandbruntime‚8021  _runtime‚8021¾dYò÷&š"ñM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15404566140416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328141557260800 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215808786 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89356517 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215808786 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55754478 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55755123 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚352127032275456 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343546351618048 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚328141539257856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145110995 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15404794357248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152743382 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15404566140416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368552168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15672872956928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326076388424428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310716537958400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343546351618048 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343546105398272 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15404794357248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚352127008335360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15673141765120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328141539257856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367800174040576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326076629113144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992âj‹óL›"öM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343546351618048 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343546105398272 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15404794357248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚352127008335360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15673141765120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328141539257856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367800174040576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326076629113144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15404566140416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328141557260800 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215808786 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89356517 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215808786 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55754478 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55755123 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚352127032275456 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343546351618048 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚328141539257856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145110995 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15404794357248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152743382 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15404566140416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368552168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15672872956928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326076388424428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310716537958400¨¢DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354144323451904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153611613 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217038441 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345514972306432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145935844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345514726099968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚330020116519936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217064111 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚327945250719488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15764845149184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89863912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15449668676996 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15494855786496®‹Sþ$ª"ìO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345514726099968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354144323451904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153611613 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217038441 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345514972306432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145935844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345514726099968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚330020116519936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217064111 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚327945250719488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15764845149184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89863912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15449668676996 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15494855786496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15494855786496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89863914 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 Jô N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 370675724 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153611611 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256«p a2«"- _wandbruntime‚8068  _runtime‚8068=©K™÷&¬"ñM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15766436805120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153629332 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145952703 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153629332 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15451724725952 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153629330 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚345554901738496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚684 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217089688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145953322 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 370719020 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚312531643811840 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15766703516160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884245975361223e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15496465277440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56078436 trainer/global_step‚8349 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚330058454464000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 370719020 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89874269 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚345555147958272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚345555147958272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚330058436461056 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15496693494272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89874267 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15451501076596 _runtime‚8068.286679288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256§>jü&­"öM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153629330 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚345554901738496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚684 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217089688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145953322 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 370719020 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚312531643811840 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15766703516160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884245975361223e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15496465277440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56078436 trainer/global_step‚8349 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚330058454464000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 370719020 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89874269 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚345555147958272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚345555147958272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚330058436461056 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15496693494272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89874267 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15451501076596 _runtime‚8068.286679288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15766436805120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153629332 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145952703 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153629332 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15451724725952¬ì’ÊÊ2®"- _wandbruntime‚8068  _runtime‚8068ܬM[X¯"Ò 1loss/SRqqt1/assignment_loss‚1.480694381455923 0loss/FBt1/assignment_loss‚0.2784252459990866 1loss/SRqqt2/detection_loss‚1.2061816736089963 2loss/SRqqt2/assignment_loss‚0.9132659026180152 /loss/FRt2/detection_loss‚1.8279248528182506 /loss/FBt2/detection_loss‚0.5388315630605421 &loss/total_loss‚41.328599511247695 % _timestamp‚1.7884245975380793e+09 1loss/SRqqt3/assignment_loss‚1.329632295738356 0loss/FBt2/assignment_loss‚0.1288515549707904 .loss/FBt3/assignment_loss‚0.14981849049218 1loss/SRqqt4/detection_loss‚2.0421408594779247 epoch‚6 2loss/SRqqt4/assignment_loss‚1.6162378161020998 1loss/SRqqt3/detection_loss‚1.5902604773396352 /loss/FBt4/detection_loss‚2.7437803506851197 trainer/global_step‚8349 0loss/FRt2/assignment_loss‚1.1765988296932643 1loss/FBt4/assignment_loss‚0.29614280657842756 /loss/FBt1/detection_loss‚2.5725476609336004 _runtime‚8068.287934603 _step‚685 0loss/FRt1/assignment_loss‚3.4393057312284196 /loss/FRt3/assignment_loss‚2.956696384896835 0loss/FRt4/assignment_loss‚3.3889753059907393 .loss/FRt1/detection_loss‚2.639256004350526 .loss/FRt3/detection_loss‚2.285343859344721 0loss/SRqqt1/detection_loss‚2.433726554286891 .loss/FRt4/detection_loss‚2.116009794704255 /loss/FBt3/detection_loss‚2.1779511148730912%?Ï,]°"× 0loss/FRt1/assignment_loss‚3.4393057312284196 /loss/FRt3/assignment_loss‚2.956696384896835 0loss/FRt4/assignment_loss‚3.3889753059907393 .loss/FRt1/detection_loss‚2.639256004350526 .loss/FRt3/detection_loss‚2.285343859344721 0loss/SRqqt1/detection_loss‚2.433726554286891 .loss/FRt4/detection_loss‚2.116009794704255 /loss/FBt3/detection_loss‚2.1779511148730912 1loss/SRqqt1/assignment_loss‚1.480694381455923 0loss/FBt1/assignment_loss‚0.2784252459990866 1loss/SRqqt2/detection_loss‚1.2061816736089963 2loss/SRqqt2/assignment_loss‚0.9132659026180152 /loss/FRt2/detection_loss‚1.8279248528182506 /loss/FBt2/detection_loss‚0.5388315630605421 &loss/total_loss‚41.328599511247695 % _timestamp‚1.7884245975380793e+09 1loss/SRqqt3/assignment_loss‚1.329632295738356 0loss/FBt2/assignment_loss‚0.1288515549707904 .loss/FBt3/assignment_loss‚0.14981849049218 1loss/SRqqt4/detection_loss‚2.0421408594779247 epoch‚6 2loss/SRqqt4/assignment_loss‚1.6162378161020998 1loss/SRqqt3/detection_loss‚1.5902604773396352 /loss/FBt4/detection_loss‚2.7437803506851197 trainer/global_step‚8349 0loss/FRt2/assignment_loss‚1.1765988296932643 1loss/FBt4/assignment_loss‚0.29614280657842756 /loss/FBt1/detection_loss‚2.5725476609336004 _runtime‚8068.287934603 _step‚685­—›Çž±"‚(:… —ãäÔ‘»x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S”0`o²"‚(:ä —ãäÔ×Þãz disk.dm-1.out‚ 393.8515625 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.out‚0 network.sent‚ 2.0535318e+07 network.recv‚ 2.845291e+06 disk.rbd2.in‚0 disk.sda3.out‚ 393.8515625 disk.nvme0n1.in‚5.3125$ memory_percent‚97.96197414398193( proc.memory.availableMB‚ 333.91015625# proc.memory.rssMB‚ 2851.32421875 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 cpu‚25.27376935357875 disk.sda3.in‚7.3125* proc.memory.percent‚17.403101921081543 disk.rbd3.in‚0! disk.dm-0.out‚36063.462890625 disk.sda1.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚36063.462890625) disk./.usagePercent‚73.71853867090064 disk.rbd0.out‚0& disk.nvme0c0n1.out‚36063.462890625! disk.rbd2.out‚237.65087890625 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0 disk.sda.out‚ 393.890625 proc.cpu.threads‚17 disk.dm-0.in‚5.3125 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd1.out‚ 4.04931640625% disk./.usageGB‚1098.4594993591309 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125ÝÈ+Ž³"‚(:ƒ ¦ãäÔü­x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.37' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€sƒÐt´"‚(:é ¦ãäÔùœôz disk.rbd1.out‚ 4.04931640625& disk.nvme0n1.out‚36162.78857421875# disk.dm-0.out‚36162.78857421875$ memory_percent‚97.96059131622314 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda.out‚ 394.4296875 disk.nvme0n1.in‚5.3125 cpu‚25.27335470611493 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda.in‚ 7.36328125* proc.memory.percent‚17.403101921081543% disk./.usageGB‚1098.4594421386719 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125) disk./.usagePercent‚73.71853483078762 disk.sda2.in‚0 disk.sda3.in‚7.3125( proc.memory.availableMB‚ 334.13671875! disk.rbd2.out‚239.22509765625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚36162.78857421875 network.recv‚ 2.852282e+06 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 394.390625# proc.memory.rssMB‚ 2851.32421875 network.sent‚ 2.0643588e+07 disk.sda3.out‚ 394.390625JŠ`UŽµ"‚(:ƒ µãäÔÅë’x gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.786' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:µ–mv¶"‚(:ë µãäÔ¾µ{ disk.rbd2.in‚0# proc.memory.rssMB‚ 2851.32421875 disk.sda3.out‚ 394.87109375 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 394.87109375 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 394.91015625 disk.rbd1.out‚ 4.04931640625# disk.dm-0.out‚36163.06201171875( disk.nvme0c0n1.out‚36163.06201171875 disk.nvme0n1.in‚5.3125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.403101921081543% disk./.usageGB‚1098.4594230651855 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125& disk.nvme0n1.out‚36163.06201171875) disk./.usagePercent‚73.71853355074995 disk.drbd1777.out‚0$ memory_percent‚97.96056747436523 network.recv‚ 2.857204e+06 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125! disk.rbd2.out‚239.28759765625 disk.sda3.in‚7.3125 disk.rbd1.in‚0 network.sent‚ 2.0649979e+07 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0& proc.memory.availableMB‚ 334.140625 cpu‚25.2729699666768P&é2·"- _wandbruntime‚8113  _runtime‚8113±£<:Ǹ"Á ' lr-AdamW/pg1‚0.00020703315395030814 ' lr-AdamW/pg2‚0.00020703315395030814 trainer/global_step‚8399 # _timestamp‚1.78842464279544e+09 _runtime‚8113.545418986 _step‚686ô‡ô&̹"Æ trainer/global_step‚8399 # _timestamp‚1.78842464279544e+09 _runtime‚8113.545418986 _step‚686 ' lr-AdamW/pg1‚0.00020703315395030814 ' lr-AdamW/pg2‚0.00020703315395030814®†¤j—2º"- _wandbruntime‚8113  _runtime‚8113÷1£Ò†»"çO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218319343 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90381662 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372061217579520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329852007183496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218345013 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90381664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15541525897352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15586754923520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚331937013723136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154497561 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚314310481286144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚314310464246784 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 _runtime‚8113.546443055 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347523522440192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154497563 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 372816904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218345013 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDevy£u*giceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚347523522440192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15586526720000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚356202808264704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281404416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146777268 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚347523768646656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚347523768646656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 trainer/global_step‚8399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56396252 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154497561 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15586754923520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218319343 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚356202832204800 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15858103970304 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15586526720000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15541302248204 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146777916 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 372842576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15858385374720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884246427962604e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚331936995720192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154497563 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚331936995720192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚687 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397–§ûðò'¼"ìO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚687 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218319343 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90381662 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372061217579520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329852007183496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218345013 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90381664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15541525897352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15586754923520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚331937013723136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154497561 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚314310481286144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚314310464246784 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 _runtime‚8113.546443055 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347523522440192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154497563 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 372816904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218345013 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚347523522440192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15586526720000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚356202808264704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281404416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146777268 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚347523768646656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚347523768646656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 trainer/global_step‚8399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56396252 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154497561 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15586754923520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218319343 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚356202832204800 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15858103970304 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15586526720000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15541302248204 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146777916 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 372842576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15858385374720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884246427962604e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚331936995720192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154497563 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚331936995720192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2¯MV ò2½"- _wandbruntime‚8114  _runtime‚8114<ˆ’÷&¾"ñM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331975351667200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15588592631296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 372860174 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329890142041160 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚347563698078720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90392019 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146794865 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146794232 _step‚688 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 _runtime‚8114.408960846 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 372885872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15543358297160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15543134647804 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8399 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 372860174 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154515282 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚347563944298496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56402846 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372103963595264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15859985634304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚331975351667200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15588364414464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218370590 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884246436577756e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 218344894 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚347563944298496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372104262635008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚314346766704640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154515280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚331975333664256 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56402215 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚356244001901056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154515280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 372885872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚356243977960960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15860260733952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218370590 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15588592631296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15588364414464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154515282 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329889901352444 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146794232 _step‚688 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 _runtime‚8114.408960846 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 372885872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15543358297160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15543134647804 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8399 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 372860174 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154515282 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚347563944298496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56402846 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372103963595264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15859985634304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚331975351667200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15588364414464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218370590 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884246436577756e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 218344894 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚347563944298496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak?*¡‚3428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372104262635008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚314346766704640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154515280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚331975333664256 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56402215 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚356244001901056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154515280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 372885872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚356243977960960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15860260733952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218370590 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800°>»±ñ2À"- _wandbruntime‚8114  _runtime‚8114¿ŒÌYÁ"Ó 0loss/FBt4/assignment_loss‚1.5919230256986339 &loss/total_loss‚41.391743679982206 epoch‚6 1loss/SRqqt1/assignment_loss‚1.758171740617301 2loss/SRqqt3/assignment_loss‚1.2183456215160815 1loss/SRqqt3/detection_loss‚1.8765579055656085 1loss/SRqqt2/assignment_loss‚0.934734647492102 0loss/SRqqt2/detection_loss‚0.782484196126461 0loss/FBt2/detection_loss‚0.49393331238793003 0loss/FRt1/assignment_loss‚3.3476347891907943 .loss/FRt3/detection_loss‚2.734155335136362 0loss/SRqqt1/detection_loss‚2.337500120336945 0loss/FRt4/assignment_loss‚3.0340278523939626 0loss/FRt3/assignment_loss‚2.5732539473353206 /loss/FRt1/detection_loss‚2.6005159796852815 /loss/FRt2/detection_loss‚1.7764194111029308 -loss/FBt4/detection_loss‚2.68122775554657 % _timestamp‚1.7884246436602087e+09 0loss/FBt1/assignment_loss‚0.2552855962944291 0loss/FBt2/assignment_loss‚0.1937823880558513 0loss/FBt3/assignment_loss‚0.3134034111513756 /loss/FRt4/detection_loss‚2.0166168415143297 1loss/SRqqt4/detection_loss‚1.9282072993616264 trainer/global_step‚8399 _runtime‚8114.410092207 .loss/FBt1/detection_loss‚1.978343497623097 /loss/FBt3/detection_loss‚2.2127938628196717 _step‚689 0loss/FRt2/assignment_loss‚1.3508601983388264 2loss/SRqqt4/assignment_loss‚1.4015649446907144¹Ñ{^Â"Ø 0loss/FBt1/assignment_loss‚0.2552855962944291 0loss/FBt2/assignment_loss‚0.1937823880558513 0loss/FBt3/assignment_loss‚0.3134034111513756 /loss/FRt4/detection_loss‚2.0166168415143297 1loss/SRqqt4/detection_loss‚1.9282072993616264 trainer/global_step‚8399 _runtime‚8114.410092207 .loss/FBt1/detection_loss‚1.978343497623097 /loss/FBt3/detection_loss‚2.2127938628196717 _step‚689 0loss/FRt2/assignment_loss‚1.3508601983388264 2loss/SRqqt4/assignment_loss‚1.4015649446907144 0loss/FBt4/assignment_loss‚1.5919230256986339 &loss/total_loss‚41.391743679982206 epoch‚6 1loss/SRqqt1/assignment_loss‚1.758171740617301 2loss/SRqqt3/assignment_loss‚1.2183456215160815 1loss/SRqqt3/detection_loss‚1.8765579055656085 1loss/SRqqt2/assignment_loss‚0.934734647492102 0loss/SRqqt2/detection_loss‚0.782484196126461 0loss/FBt2/detection_loss‚0.49393331238793003 0loss/FRt1/assignment_loss‚3.3476347891907943 .loss/FRt3/detection_loss‚2.734155335136362 0loss/SRqqt1/detection_loss‚2.337500120336945 0loss/FRt4/assignment_loss‚3.0340278523939626 0loss/FRt3/assignment_loss‚2.5732539473353206 /loss/FRt1/detection_loss‚2.6005159796852815 /loss/FRt2/detection_loss‚1.7764194111029308 -loss/FBt4/detection_loss‚2.68122775554657 % _timestamp‚1.7884246436602087e+09±÷éâ­Ã"‚(:… ÄãäÔåð¡x gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.584' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d‘ú·xÄ"‚(:í ÄãäÔ‰¿§{) disk./.usagePercent‚73.71853355074995 disk.sda1.out‚0 disk.sda3.in‚7.3125 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚36262.59619140625 cpu‚25.27252909317947( disk.nvme0c0n1.out‚36262.59619140625 disk.rbd1.out‚ 4.04931640625* proc.memory.percent‚17.403101921081543 disk.sda2.out‚ 0.0390625 disk.nvme0c0n1.in‚5.3125& disk.nvme0n1.out‚36262.59619140625 disk.dm-0.in‚5.3125! disk.rbd2.out‚239.31103515625 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0( proc.memory.availableMB‚ 335.17578125# proc.memory.rssMB‚ 2851.32421875% disk./.usageGB‚1098.4594230651855 disk.sda2.in‚0 disk.rbd4.out‚0 disk.sda3.out‚ 395.44921875 disk.nvme0n1.in‚5.3125 network.recv‚ 2.86074e+06 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 395.44921875 disk.sda.out‚ 395.48828125$ memory_percent‚97.95424938201904 network.sent‚ 2.0655352e+07ßn¤æ‹Å"‚(:€ ÓãäÔ턘x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.5' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™ùStÆ"‚(:é ÓãäÔ…à‡{ disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 395.8671875 disk.dm-0.in‚5.3125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 cpu‚25.272121485309444 proc.cpu.threads‚17) disk./.usagePercent‚73.71852894261434( disk.nvme0c0n1.out‚36361.65087890625 disk.dm-1.in‚7.3125 disk.rbd2.in‚0$ memory_percent‚97.95563220977783( proc.memory.availableMB‚ 334.94921875# proc.memory.rssMB‚ 2851.32421875 disk.sda1.out‚0 network.sent‚ 2.0763787e+07% disk./.usageGB‚1098.4593544006348 disk.rbd3.in‚0 disk.nvme0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.dm-1.out‚ 395.828125 disk.rbd2.out‚239.4013671875 disk.sda3.out‚ 395.828125& disk.nvme0n1.out‚36361.65087890625 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.sda3.in‚7.3125 disk.rbd1.in‚0* proc.memory.percent‚17.403101921081543 network.recv‚ 2.868072e+06 disk.drbd1777.out‚0 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 4.04931640625# disk.dm-0.out‚36361.65087890625È7mÇ"‚(:… âãäÔì» x gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7596 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n ì¯pÈ"‚(:å âãäÔ‡€Âz disk.sda3.out‚ 396.4921875 disk.sda2.in‚0 disk.rbd0.in‚0# memory_percent‚97.9562759399414% proc.memory.availableMB‚ 334.84375 network.recv‚ 2.872113e+06) disk./.usagePercent‚73.71856836777461 disk.sda.out‚ 396.53125 cpu‚25.271686650034752 disk.dm-1.out‚ 396.4921875 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.dm-0.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚36461.63720703125 disk.rbd0.out‚0# proc.memory.rssMB‚ 2851.32421875* proc.memory.percent‚17.403101921081543 network.sent‚ 2.0769655e+07% disk./.usageGB‚1098.4599418640137 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.04931640625& disk.nvme0n1.out‚36461.63720703125 disk.rbd2.out‚239.4169921875 proc.cpu.threads‚17 disk.dm-1.in‚7.3125( disk.nvme0c0n1.out‚36461.63720703125 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.sda2.out‚ 0.0390625œB{a2É"- _wandbruntime‚8159  _runtime‚8159ƒº|ÈÊ" _step‚690 ' lr-AdamW/pg1‚0.00020198559199592354 ' lr-AdamW/pg2‚0.00020198559199592354 trainer/global_step‚8449 $ _timestamp‚1.788424688358126e+09 _runtime‚8159.108041457,¸>ÍË"Ç ' lr-AdamW/pg2‚0.00020198559199592354 trainer/global_step‚8449 $ _timestamp‚1.788424688358126e+09 _runtime‚8159.108041457 _step‚690 ' lr-AdamW/pg1‚0.00020198559199592354²Ûо2Ì"- _wandbruntime‚8159  _runtime‚8159Âhå-í'Í"çO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15678654060544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333853892923392 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147624130 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15678425857024 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15632935819412 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚668 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15678654060544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90899412 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358261293077504 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155383513 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15633159468560 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349532318780416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333853910926336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219625915 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 trainer/global_step‚8449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚333853892923392 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219600245 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155383513 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56724048 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚374213287801344 _runtime‚8159.109054125 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219600245 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349532318780416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 375009428 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚331758763647504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚316125604178944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15951970783744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374983756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155383511 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56724716 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚670 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147623460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219625915 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358261317017600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349532564986880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15678425857024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚331758522958996 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333853910926336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚316125587139584 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155383511 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚374212986651136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349532564986880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15951693573632 _step‚691 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15632935819412 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚668 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15678654060544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90899412 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358261293077504 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155383513 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15633159468560 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349532318780416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333853910926336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219625915 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 trainer/global_step‚8449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚333853892923392 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219600245 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155383513 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56724048 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚374213287801344 _runtime‚8159.109054125 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219600245 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349532318780416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 375009428 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚331758763647504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚316125604178944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15951970783744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374983756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155383511 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56724716 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚670 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147623460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219625915 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358261317017600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349532564986880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15678425857024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚331758522958996 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333853910926336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚316125587139584 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155383511 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚374212986651136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349532564986880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15951693573632 _step‚691 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358302486713856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15953565847552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚316161889597440 XIDeviceStatsMonitor.on_train_Rã”%batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333892230867456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 375052724 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15634991868368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155401230 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 375027026 % _timestamp‚1.7884246892234116e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚331796898505168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 trainer/global_step‚8449 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90909767 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚316161906636800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358302462773760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚374256319272448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155401230 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚374256028621312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219625796 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 375052724 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15680491768320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15680263551488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15680263551488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349572740638720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147640007 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155401232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349572740638720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333892230867456 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 219651492 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15634768219012 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147640644 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚692 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56730875 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚331796657816452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155401232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333892248870400 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15680491768320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0\DÿÆü&Ñ"öM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚316161906636800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358302462773760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚374256319272448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155401230 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚374256028621312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219625796 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 375052724 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15680491768320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15680263551488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15680263551488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349572740638720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147640007 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155401232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349572740638720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333892230867456 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 219651492 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15634768219012 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147640644 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚692 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56730875 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚331796657816452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155401232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333892248870400 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15680491768320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358302486713856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15953565847552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚316161889597440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333892230867456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 375052724 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15634991868368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155401230 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 375027026 % _timestamp‚1.7884246892234116e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚331796898505168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 trainer/global_step‚8449 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90909767 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0´‹„º2Ò"- _wandbruntime‚8159  _runtime‚8159+óíïYÓ"Ó /loss/FRt2/assignment_loss‚1.147261098319409 1loss/FBt4/assignment_loss‚0.39463932275409913 .loss/FRt3/detection_loss‚2.141538487604031 /loss/SRqqt4/detection_loss‚1.97714118544872 .loss/FBt1/detection_loss‚2.130143863814218 /loss/FBt3/detection_loss‚3.2739148288965225 % _timestamp‚1.7884246892252514e+09 /loss/FRt2/detection_loss‚1.4376696473827548 .loss/FRt4/detection_loss‚2.247919436644998 /loss/FBt2/detection_loss‚0.4260492811863048 _step‚693 2loss/SRqqt3/assignment_loss‚1.6425997042202984 /loss/FRt1/detection_loss‚2.8716354502571955 trainer/global_step‚8449 0loss/FRt4/assignment_loss‚3.2218053863973033 1loss/FBt2/assignment_loss‚0.09488174784837074 0loss/FRt1/assignment_loss‚3.2644686798254647 /loss/FRt3/assignment_loss‚2.622704009597118 2loss/SRqqt1/assignment_loss‚1.7233277659963917 2loss/SRqqt4/assignment_loss‚1.6302170113732035 /loss/FBt3/assignment_loss‚0.302622516581323 1loss/SRqqt2/detection_loss‚0.9958227929877647 _runtime‚8159.975110836 0loss/SRqqt1/detection_loss‚2.337931172267811 epoch‚6 2loss/SRqqt2/assignment_loss‚1.0064477636076419 1loss/FBt1/assignment_loss‚0.19221263019634144 1loss/SRqqt3/detection_loss‚2.0671277537264605 /loss/FBt4/detection_loss‚1.8880127337243822 %loss/total_loss‚41.03809427065813Ÿiì^Ô"Ø 2loss/SRqqt4/assignment_loss‚1.6302170113732035 /loss/FBt3/assignment_loss‚0.302622516581323 1loss/SRqqt2/detection_loss‚0.9958227929877647 _runtime‚8159.975110836 0loss/SRqqt1/detection_loss‚2.337931172267811 epoch‚6 2loss/SRqqt2/assignment_loss‚1.0064477636076419 1loss/FBt1/assignment_loss‚0.19221263019634144 1loss/SRqqt3/detection_loss‚2.0671277537264605 /loss/FBt4/detection_loss‚1.8880127337243822 %loss/total_loss‚41.03809427065813 /loss/FRt2/assignment_loss‚1.147261098319409 1loss/FBt4/assignment_loss‚0.39463932275409913 .loss/FRt3/detection_loss‚2.141538487604031 /loss/SRqqt4/detection_loss‚1.97714118544872 .loss/FBt1/detection_loss‚2.130143863814218 /loss/FBt3/detection_loss‚3.2739148288965225 % _timestamp‚1.7884246892252514e+09 /loss/FRt2/detection_loss‚1.4376696473827548 .loss/FRt4/detection_loss‚2.247919436644998 /loss/FBt2/detection_loss‚0.4260492811863048 _step‚693 2loss/SRqqt3/assignment_loss‚1.6425997042202984 /loss/FRt1/detection_loss‚2.8716354502571955 trainer/global_step‚8449 0loss/FRt4/assignment_loss‚3.2218053863973033 1loss/FBt2/assignment_loss‚0.09488174784837074 0loss/FRt1/assignment_loss‚3.2644686798254647 /loss/FRt3/assignment_loss‚2.622704009597118 2loss/SRqqt1/assignment_loss‚1.7233277659963917µÈÏ‹™Õ"‚(:Ž ñãäÔ–ðŒy gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.72' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.343999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Âà03uÖ"‚(:ê ñãäÔøöÔz disk.sda.in‚ 7.36328125 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda.out‚ 397.02734375 disk.sda3.in‚7.3125# proc.memory.rssMB‚ 2851.32421875 disk.sda1.out‚0 disk.sda3.out‚ 396.98828125 proc.cpu.threads‚17 network.sent‚ 2.0775945e+07 disk.rbd2.out‚241.1318359375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚36560.73291015625' proc.memory.availableMB‚ 334.7734375) disk./.usagePercent‚73.71848516532597% disk./.usageGB‚1098.4587020874023& disk.nvme0n1.out‚36560.73291015625 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.rbd3.in‚0 network.recv‚ 2.876704e+06 disk.drbd1777.in‚ 1.03515625* proc.memory.percent‚17.403101921081543 disk.rbd1.in‚0 disk.rbd1.out‚ 4.04931640625 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0$ memory_percent‚97.95670509338379 disk.nvme0n1.in‚5.3125 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 cpu‚25.271310029281# disk.dm-0.out‚36560.73291015625 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 396.98828125æö¢¨Ž×"‚(:ƒ €ääÔ½Ž°x gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐÌøÛhØ"‚(:Ý €ääÔü³éz disk.rbd3.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚0' disk.nvme0c0n1.out‚36660.1337890625& proc.memory.availableMB‚ 332.265625) proc.memory.percent‚17.41769313812256 network.sent‚ 2.0883626e+07 disk.dm-1.in‚7.3125$ memory_percent‚97.97201156616211 network.recv‚ 2.8831e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚5.3125" disk.dm-0.out‚36660.1337890625$ disk./.usageGB‚1098.458381652832 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.rbd1.out‚ 4.04931640625% disk.nvme0n1.out‚36660.1337890625 proc.cpu.threads‚17 disk.rbd2.out‚241.1474609375 disk.nvme0n1.in‚5.3125 disk.dm-1.out‚ 397.53125 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚25.272611241342155 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 397.5703125 disk.sda3.out‚ 397.53125 disk.sda2.out‚ 0.0390625# proc.memory.rssMB‚ 2853.71484375( disk./.usagePercent‚73.7184636606931 disk.drbd1777.out‚0†X-³ŽÙ"‚(:ƒ ääÔ½–­x gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0irp?qÚ"‚(:æ ääÔ’«–z" disk.dm-0.out‚36759.1611328125 disk.rbd4.out‚0$ disk./.usageGB‚1098.458381652832 disk.rbd2.in‚0' disk.nvme0c0n1.out‚36759.1611328125 disk.sda3.in‚7.3125 disk.sda3.out‚ 398.08984375 disk.rbd1.in‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 398.12890625 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 398.08984375 disk.sda1.out‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 323.26171875 cpu‚25.28066765409543 disk.dm-0.in‚5.3125 disk.rbd1.out‚ 4.04931640625 proc.cpu.threads‚17 network.sent‚ 2.0889701e+07 network.recv‚ 2.887745e+06 disk.sda2.in‚0 disk.rbd2.out‚241.2724609375 disk.drbd1777.in‚ 1.03515625 disk.nvme0n1.in‚5.3125% disk.nvme0n1.out‚36759.1611328125$ memory_percent‚98.02696704864502 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125" proc.memory.rssMB‚ 2862.7578125) proc.memory.percent‚17.47288703918457( disk./.usagePercent‚73.7184636606931TJ6ŽÛ"‚(:ƒ žääÔ©¦x gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£kÆ’jÜ"‚(:ß žääÔÐÈ»{ disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125$ memory_percent‚98.02694320678711& proc.memory.availableMB‚ 323.265625 proc.cpu.threads‚17) disk./.usagePercent‚73.71843882796229 disk.sda3.in‚7.3125 network.sent‚ 2.089561e+07 network.recv‚ 2.891514e+06 disk.rbd2.out‚241.2880859375 disk.rbd4.in‚0" proc.memory.rssMB‚ 2862.7578125 disk.rbd0.out‚0 disk.sda1.out‚0! disk.dm-0.out‚36759.662109375 disk.sda3.out‚ 398.6328125 disk.dm-0.in‚5.3125) proc.memory.percent‚17.47288703918457 cpu‚25.288348247637067 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.04931640625 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 398.6328125 disk.rbd0.in‚0 disk.rbd4.out‚0& disk.nvme0c0n1.out‚36759.662109375 disk.sda.out‚ 398.671875% disk./.usageGB‚1098.4580116271973 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚5.3125$ disk.nvme0n1.out‚36759.662109375ÜR/ÚÝ"‚(:‚ ­ääÔÑƵx gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.46' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.892 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ„nkÞ"‚(:à ­ääÔ±‡‹| disk.dm-1.out‚ 399.046875 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125 cpu‚25.29570191358914 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd4.out‚0& disk.nvme0c0n1.out‚36858.763671875$ memory_percent‚98.02632331848145' proc.memory.availableMB‚ 323.3671875 network.recv‚ 2.896451e+06% disk./.usageGB‚1098.4578971862793 disk.sda2.out‚ 0.0390625) disk./.usagePercent‚73.71843114773625 disk.rbd0.in‚0 disk.sda3.out‚ 399.046875 disk.rbd1.out‚ 4.04931640625) proc.memory.percent‚17.47288703918457 network.sent‚ 2.0901901e+07 disk.sda1.out‚0! disk.dm-0.out‚36858.763671875" proc.memory.rssMB‚ 2862.7578125 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚5.3125 disk.dm-1.in‚7.3125! disk.rbd2.out‚241.37841796875 disk.sda.in‚ 7.36328125 disk.sda.out‚ 399.0859375$ disk.nvme0n1.out‚36858.763671875Ÿ Ž¹2ß"- _wandbruntime‚8224  _runtime‚8224BÖYmo à"ÍT _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚376178281103872 _runtime‚8224.057563507 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚335671522838528 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 220388688 ^QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚90898432 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 $ _timestamp‚1.788424753307144e+09 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31168 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚351395185347072 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 6893338624 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 156363441 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚23 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚15723644505600 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31397 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚91483618 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 168899072 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚15677997828524 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚57079429 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚360181003426816 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 6893338624 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚15723644505600 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚335671522838528 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 189713920 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 6893338624 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6252512256 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1261 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 144958976 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚561 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6739344896ÞÑÀ¯X*á"ÒT OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 376752131 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚317846716062720 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚333524546069844 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚561 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 156363441 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 220388688 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚15997422636032 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚262 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 376731928 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 6893338624 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6252512256 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1261 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 144958976 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚561 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6739344896 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚376178281103872 _runtime‚8224.057563507 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚335671522838528 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 220388688 ^QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚90898432 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 $ _timestamp‚1.788424753307144e+09 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31168 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚351395185347072 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 6893338624 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 156363441 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚23 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚15723644505600 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31397 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚91483618 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 168899072 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚15677997828524 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚57079429 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚360181003426816 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 6893338624 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚15723644505600 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚335671522838528 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 189713920 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31397 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 537294673 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31168 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚15723865589760 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1261 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚351421529218048 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚335697681631232 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚335697681631232 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 156378377 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚15678389005420 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚15678221184020 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚15724037300736 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚376206257789952 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚262 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 376775343 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 220396964 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 184860760 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20203 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚380 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 189713920 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 376755140 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚91492584 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 541892608 $ _timestamp‚1.788424754388656e+09 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 156378379 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20201 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚15724037300736 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 541892608 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20203 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 148577736 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚91492582 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 167821400 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31397 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚335697663628288 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚360208442358784 \ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚90898432 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚333549875056748 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6522528073 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 549453824 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚262 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚351421718931968 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3428 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20201 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚15997670472192 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚262 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚351421718931968 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3428 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20201 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚15997670472192 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31397 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 537294673 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31168 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚15723865589760 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1261 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚351421529218048 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚335697681631232 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚335697681631232 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 156378377 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚15678389005420 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚15678221184020 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚15724037300736 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚376206257789952 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚262 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 376775343 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 220396964 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 184860760 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20203 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚380 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 189713920 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 376755140 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚91492584 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 541892608 $ _timestamp‚1.788424754388656e+09 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 156378379 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20201 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚15724037300736 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 541892608 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20203 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 148577736 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚91492582 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 167821400 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31397 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚335697663628288 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚360208442358784 \ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚90898432·ÊD¨å"‚(:… ¼ääÔß²Óy gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G±©¤mæ"‚(:â ¼ääÔÜà»z disk.dm-1.out‚399.6875 disk.dm-0.in‚5.3125# disk.dm-0.out‚36958.43310546875 disk.sda2.out‚ 0.0390625 disk.sda3.in‚7.3125$ memory_percent‚98.02813529968262) disk./.usagePercent‚73.71846084461022 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 399.7265625) proc.memory.percent‚17.47288703918457 disk.sda1.out‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2862.7578125 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚36958.43310546875 disk.rbd3.in‚0 disk.dm-1.in‚7.3125 network.recv‚ 2.899784e+06$ disk./.usageGB‚1098.458339691162 disk.rbd0.in‚0' proc.memory.availableMB‚ 323.0703125 disk.rbd1.out‚ 4.04931640625& disk.nvme0n1.out‚36958.43310546875 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 proc.cpu.threads‚17 network.sent‚ 2.0907205e+07 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 disk.sda.in‚ 7.36328125 disk.sda3.out‚399.6875 cpu‚25.303466816816016 disk.rbd1.in‚0! disk.rbd2.out‚241.39404296875 disk.sda1.in‚ 0.05078125ˆÎù$2ç"- _wandbruntime‚8239  _runtime‚8239–w¦^\è"Õ¦ >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 (particle/accuracy‚0.8543370962142944 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 *jet/accuracy_1_of_6‚0.9807692170143127 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 trainer/global_step‚8462 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9918062090873718 particle/accuracy_1_of_3‚1 jet/accuracy_2_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12531571090221405 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.475970059633255 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_3_of_7‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15038025379180908 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06427130103111267 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.003908720798790455 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3659428656101227 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16866295039653778 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5378663539886475 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974586963653564 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_6_of_7‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18085484206676483 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23362717032432556 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3437499701976776 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5083141326904297 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5144681930541992 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3022626042366028 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5017766952514648 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3010416626930237 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5023621320724487 +jet/accuracy_2_of_3‚0.22475987672805786 particle/accuracy_4_of_7‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06796718388795853 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5307618379592896 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09554383903741837 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16577477753162384 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_3‚0.8562027812004089 particle/accuracy_5_of_5‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3333333432674408 jet/accuracy_3_of_7‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5048756003379822 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3452441394329071 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4435344934463501 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0163690485060215 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19758538901805878 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4482003152370453 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.203375443816185 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3828543722629547 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2075757533311844 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 _step‚696 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9901342391967773 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02157009020447731 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10302487015724182 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9886882901191711 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15446658432483673 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08153459429740906 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3759258985519409 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5033824443817139 particle/accuracy_2_of_3‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_7_of_7‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9906136393547058 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4687969982624054 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.48227378726005554 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4753374457359314 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9800435304641724 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14430974423885345 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4418919086456299 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.2348484843969345 jet/accuracy_7_of_7‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6703078746795654 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998635470867157 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9856321215629578 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.135589599609375 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5201483964920044 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005442176945507526 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023220686241984367 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13203714787960052 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9933333396911621 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15394380688667297 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 jet/accuracy_5_of_6‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.179021954536438 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4795191287994385 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2759259343147278 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3883040249347687 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28796297311782837 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15146197378635406 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011386143043637276 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.40824151039123535 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5054328441619873 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4905933141708374 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24310344457626343 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 *loss/val_total_loss‚42.027465317412556 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4226583242416382 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2530299425125122 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.15685661137104034 )jet/accuracy_2_of_5‚0.464164137840271 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4062500298023224 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 epoch‚6 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1896396279335022 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.43519726395606995 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_3‚0.020875709131360054 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25148093700408936 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,particle/sensitivity‚0.30913296341896057 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.012717309407889843 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 jet/accuracy_5_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13970695436000824 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5455325245857239 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877768158912659 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10265166312456131 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4058009684085846 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3988819420337677 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.005747126415371895 +jet/accuracy_4_of_6‚0.03846153989434242 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8157379627227783 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15492254495620728 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5071566700935364 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23628206551074982 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5002976059913635 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_5‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9905951619148254 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5119886994361877 7"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.46575728058815 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4347890019416809 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7377476692199707 *jet/accuracy_2_of_6‚0.5128205418586731 +jet/accuracy_3_of_6‚0.11538461595773697 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.44012027978897095 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.48666664958000183 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2579365372657776 particle/accuracy_5_of_7‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48818090558052063 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.35737377405166626 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 *jet/accuracy_1_of_5‚0.9462430477142334 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.48664015531539917 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.44907405972480774 particle/accuracy_1_of_6‚1 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.49212685227394104 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9935674667358398 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4595370590686798 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.002016128972172737 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5061774253845215 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 particle/accuracy_3_of_3‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17139096558094025 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979657530784607 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.019623374566435814 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 'particle/f_score‚0.3984566330909729 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4989449977874756 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 particle/accuracy_1_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/event_purity‚0.0194968543946743 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12655234336853027 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1540469378232956 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 % _timestamp‚1.7884247685327632e+09 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.241727Ãþ650115394592 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9910692572593689 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3439355492591858 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016232451424002647 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20916196703910828 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1271856725215912 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3392857015132904 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2361525595188141 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.992527186870575 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,jet/accuracy_4_of_4‚0.002521008485928178 jet/accuracy_1_of_7‚1 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4795191287994385 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36319446563720703 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.17192982137203217 particle/accuracy_4_of_5‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 5 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19744972884655 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 particle/accuracy_3_of_6‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2083333283662796 particle/accuracy_2_of_6‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5005359053611755 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09003832191228867 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4758755564689636 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9907804727554321 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.43619412183761597 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3708333671092987 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03276016563177109 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.42344963550567627 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2514316737651825 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15038025379180908 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4404761791229248 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9898778200149536 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4612068831920624 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2923083007335663 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15360569953918457 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_1_of_2‚1 jet/accuracy_6_of_6‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_7‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05480165407061577 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.47452959418296814 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1527777761220932 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.392374724149704 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4398989677429199 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4401682913303375 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13731887936592102 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.567361056804657 2 Purity/2FRt0SRqqt*FBt/FRt_purity‚ 0.22802734375 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 +jet/accuracy_2_of_4‚0.33687055110931396 particle/accuracy_4_of_4‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4629780650138855 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6418783068656921 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04669540002942085 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9927716851234436 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20055261254310608 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09712162613868713 +particle/specificity‚0.9552409648895264 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.44771093130111694 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4353671073913574 particle/accuracy_1_of_5‚1 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4590234160423279 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3633333146572113 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16927359998226166 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15812060236930847 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07111111283302307 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14571672677993774 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.32731741666793823 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0833333358168602 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9855191111564636 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15603448450565338 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4561403691768646 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_1_of_1‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3938997685909271 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969686269760132 *jet/accuracy_1_of_1‚0.4714813232421875 +jet/accuracy_3_of_4‚0.06147806718945503 +jet/accuracy_3_of_5‚0.06621863692998886 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.696969747543335 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2349080592393875 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 *jet/accuracy_1_of_4‚0.8998509645462036 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09876543283462524 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 jet/accuracy_6_of_7‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.011243387125432491 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.36419758200645447 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0014430014416575432 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7"Purity/0FRt*SRqqt2FBt/event_purity‚0.27238929271698 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4735623896121979 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0017857142956927419 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4451976716518402 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.047600824385881424 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22190792858600616 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5229637622833252 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.987351655960083 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1606893688440323 particle/accuracy_1_of_4‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1465293914079666 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3823625147342682 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3930443823337555 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5166267156600952 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.46738284826278687 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3484848737716675 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5069444179534912 _runtime‚ 8239.28514131 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44173935055732727 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06382978707551956 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.009002725593745708 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484¿fôHé"Ú¦ ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4401682913303375 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13731887936592102 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.567361056804657 2 Purity/2FRt0SRqqt*FBt/FRt_purity‚ 0.22802734375 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 +jet/accuracy_2_of_4‚0.33687055110931396 particle/accuracy_4_of_4‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4629780650138855 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6418783068656921 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04669540002942085 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9927716851234436 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20055261254310608 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09712162613868713 +particle/specificity‚0.9552409648895264 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.44771093130111694 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4353671073913574 particle/accuracy_1_of_5‚1 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4590234160423279 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3633333146572113 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16927359998226166 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15812060236930847 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07111111283302307 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14571672677993774 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.32731741666793823 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0833333358168602 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9855191111564636 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15603448450565338 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4561403691768646 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_1_of_1‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3938997685909271 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969686269760132 *jet/accuracy_1_of_1‚0.4714813232421875 +jet/accuracy_3_of_4‚0.06147806718945503 +jet/accuracy_3_of_5‚0.06621863692998886 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.696969747543335 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2349080592393875 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 *jet/accuracy_1_of_4‚0.8998509645462036 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09876543283462524 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 jet/accuracy_6_of_7‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.011243387125432491 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.36419758200645447 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0014430014416575432 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7"Purity/0FRt*SRqqt2FBt/event_purity‚0.27238929271698 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4735623896121979 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0017857142956927419 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4451976716518402 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.047600824385881424 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22190792858600616 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5229637622833252 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.987351655960083 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1606893688440323 particle/accuracy_1_of_4‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1465293914079666 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3823625147342682 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3930443823337555 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5166267156600952 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.46738284826278687 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3484848737716675 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5069444179534912 _runtime‚ 8239.28514131 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44173935055732727 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06382978707551956 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.009002725593745708 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 (particle/accuracy‚0.8543370962142944 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 *jet/accuracy_1_of_6‚0.9807692170143127 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 trainer/global_step‚8462 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9918062090873718 particle/accuracy_1_of_3‚1 jet/accuracy_2_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12531571090221405 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.475970059633255 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_3_of_7‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15038025379180908 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06427130103111267 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.003908720798790455 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3659428656101227 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16866295039653778 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5378663539886475 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974586963653564 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_6_of_7‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18085484206676483 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23362717032432556 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3437499701976776 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5083141326904297 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5144681930541992 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3022626042366028 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5017766952514648 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3010416626930237 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5023621320724487 +jet/accuracy_2_of_3‚0.22475987672805786 particle/accuracy_4_of_7‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06796718388795853 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5307618379592896 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09554383903741837 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16577477753162384 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_3‚0.8562027812004089 particle/accuracy_5_of_5‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3333333432674408 jet/accuracy_3_of_7‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5048756003379822 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3452441394329071 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4435344934463501 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0163690485060215 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19758538901805878 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4482003152370453 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.203375443816185 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3828543722629547 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2075757533311844 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 _step‚696 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9901342391967773 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02157009020447731 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10302487015724182 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9886882901191711 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15446658432483673 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08153459429740906 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3759258985519409 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5033824443817139 particle/accuracy_2_of_3‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_7_of_7‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9906136393547058 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4687969982624054 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.48227378726005554 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4753374457359314 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9800435304641724 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14430974423885345 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.00¹NzmJ87425597012043 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4418919086456299 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.2348484843969345 jet/accuracy_7_of_7‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6703078746795654 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998635470867157 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9856321215629578 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.135589599609375 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5201483964920044 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005442176945507526 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023220686241984367 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13203714787960052 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9933333396911621 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15394380688667297 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 jet/accuracy_5_of_6‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.179021954536438 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4795191287994385 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2759259343147278 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3883040249347687 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28796297311782837 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15146197378635406 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011386143043637276 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.40824151039123535 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5054328441619873 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4905933141708374 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24310344457626343 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 *loss/val_total_loss‚42.027465317412556 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4226583242416382 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2530299425125122 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.15685661137104034 )jet/accuracy_2_of_5‚0.464164137840271 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4062500298023224 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 epoch‚6 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1896396279335022 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.43519726395606995 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_3‚0.020875709131360054 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25148093700408936 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,particle/sensitivity‚0.30913296341896057 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.012717309407889843 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 jet/accuracy_5_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13970695436000824 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5455325245857239 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877768158912659 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10265166312456131 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4058009684085846 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3988819420337677 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.005747126415371895 +jet/accuracy_4_of_6‚0.03846153989434242 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8157379627227783 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15492254495620728 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5071566700935364 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23628206551074982 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5002976059913635 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_5‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9905951619148254 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5119886994361877 7"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.46575728058815 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4347890019416809 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7377476692199707 *jet/accuracy_2_of_6‚0.5128205418586731 +jet/accuracy_3_of_6‚0.11538461595773697 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.44012027978897095 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.48666664958000183 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2579365372657776 particle/accuracy_5_of_7‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48818090558052063 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.35737377405166626 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 *jet/accuracy_1_of_5‚0.9462430477142334 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.48664015531539917 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.44907405972480774 particle/accuracy_1_of_6‚1 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.49212685227394104 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9935674667358398 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4595370590686798 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.002016128972172737 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5061774253845215 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 particle/accuracy_3_of_3‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17139096558094025 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979657530784607 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.019623374566435814 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 'particle/f_score‚0.3984566330909729 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4989449977874756 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 particle/accuracy_1_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/event_purity‚0.0194968543946743 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12655234336853027 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1540469378232956 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 % _timestamp‚1.7884247685327632e+09 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24172750115394592 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9910692572593689 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3439355492591858 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016232451424002647 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20916196703910828 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1271856725215912 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3392857015132904 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2361525595188141 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.992527186870575 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,jet/accuracy_4_of_4‚0.002521008485928178 jet/accuracy_1_of_7‚1 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4795191287994385 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36319446563720703 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.17192982137203217 particle/accuracy_4_of_5‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 5 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19744972884655 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 particle/accuracy_3_of_6‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2083333283662796 particle/accuracy_2_of_6‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5005359053611755 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09003832191228867 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4758755564689636 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9907804727554321 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.43619412183761597 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3708333671092987 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03276016563177109 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.42344963550567627 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2514316737651825 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15038025379180908 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4404761791229248 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9898778200149536 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4612068831920624 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2923083007335663 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15360569953918457 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_1_of_2‚1 jet/accuracy_6_of_6‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_7‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05480165407061577 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.47452959418296814 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1527777761220932 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.392374724149704 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4398989677429199¸a7ÂEšê"‚(: ËääÔò­©x gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.811799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 xÀawë"‚(:ì ËääÔµª×z( disk.nvme0c0n1.out‚37224.18212890625! proc.memory.rssMB‚ 2862.765625% disk./.usageGB‚1098.4590339660645# disk.dm-0.out‚37224.18212890625 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125( proc.memory.availableMB‚ 506.41796875 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 400.09765625) proc.memory.percent‚17.47293472290039 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.sda.in‚ 7.36328125 cpu‚25.305362149974258 disk.rbd1.out‚ 4.04931640625 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.sda.out‚ 400.13671875 disk.dm-0.in‚5.3125 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda1.out‚0 network.recv‚ 2.906844e+06) disk./.usagePercent‚73.71850743798146 disk.rbd1.in‚0! disk.rbd2.out‚243.09326171875 disk.rbd4.out‚0 disk.sda3.out‚ 400.09765625$ memory_percent‚96.90907001495361 network.sent‚ 2.1015328e+07& disk.nvme0n1.out‚37224.18212890625 disk.rbd2.in‚0þs•vì"‚(:… ÚääÔ´šÀx gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚100.354' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.0708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P’5>gí"‚(:Ü ÚääÔßúçz disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda3.in‚7.3125) proc.memory.percent‚17.46692657470703 cpu‚25.304869179327973 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.sda.out‚401.125 network.sent‚ 2.1139321e+07 network.recv‚ 2.914038e+06$ disk./.usageGB‚1098.458339691162 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.sda3.out‚ 401.0859375% disk.nvme0n1.out‚37237.4892578125" disk.dm-0.out‚37237.4892578125 disk.rbd4.in‚0 disk.sda2.in‚0 disk.dm-1.in‚7.3125% proc.memory.availableMB‚ 507.15625 proc.memory.rssMB‚ 2861.78125) disk./.usagePercent‚73.71846084461022 disk.rbd0.in‚0 disk.rbd4.out‚0' disk.nvme0c0n1.out‚37237.4892578125 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.rbd1.out‚ 4.04931640625 disk.dm-0.in‚5.3125 disk.rbd0.out‚0 disk.nvme0n1.in‚5.3125! disk.rbd2.out‚243.10498046875# memory_percent‚96.9045639038086 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-1.out‚ 401.0859375úDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221404472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57384194 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚337417677884416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚337417677884416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319500145835008 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 _step‚698 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚337417659881472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15795088078336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16069955912192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚335249617165962 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157210431 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378116878290432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157210429 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _runtime‚8275.518040438 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221404472 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15749454291594 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157210431 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157210429 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16070237316608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281404416 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353212747959808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 378640573 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378117183634944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252ìö$Ÿ512256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0@æÑò'ó"ìO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚335249617165962 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157210431 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378116878290432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157210429 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _runtime‚8275.518040438 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221404472 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15749454291594 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157210431 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157210429 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16070237316608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281404416 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353212747959808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 378640573 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378117183634944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221404472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57384194 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚337417677884416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚337417677884416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319500145835008 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 _step‚698 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚337417659881472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15795088078336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16069955912192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896ºŽÉjl2ô"- _wandbruntime‚8276  _runtime‚8276Û8_ö&õ"ðM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378683869 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 378658171 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚319536465332224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378683869 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157228150 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15797153989632 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221455719 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292748288 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378159920016384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221455719 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378160212764672 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8499 $ _timestamp‚1.788424805621043e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157228148 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335287511334910 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚335287752023626 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚8276.371594602 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15796925772800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16072096750080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16071827941888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚337455997825536 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149382900 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57390010 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚362088092074496 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353253169818112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 _step‚699 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15751286691402 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚337456015828480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15797153989632 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221430023 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚337456015828480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91992251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚337455997825536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57390649 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15751063042046 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149382259¶ÝÄû&ö"õM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚335287752023626 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚8276.371594602 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15796925772800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16072096750080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16071827941888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚337455997825536 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149382900 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57390010 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚362088092074496 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353253169818112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 _step‚699 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15751286691402 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚337456015828480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15797153989632 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221430023 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚337456015828480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91992251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚337455997825536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57390649 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15751063042046 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149382259 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378683869 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 378658171 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚319536465332224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378683869 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157228150 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15797153989632 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221455719 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 292748288 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378159920016384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221455719 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378160212764672 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8499 $ _timestamp‚1.788424805621043e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157228148 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335287511334910 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262»ú¾ó&2÷"- _wandbruntime‚8276  _runtime‚8276?IV%Zø"Ô 1loss/SRqqt3/detection_loss‚1.7235885752459703 %loss/total_loss‚37.95335398547823 trainer/global_step‚8499 _runtime‚8276.372617058 /loss/FBt2/assignment_loss‚0.059673810527691 0loss/FBt3/assignment_loss‚0.5505886146565899 1loss/SRqqt2/detection_loss‚1.1148502101239406 /loss/FBt3/detection_loss‚2.4363372027873993 _step‚700 0loss/FRt3/assignment_loss‚2.2849216808875403 2loss/SRqqt1/assignment_loss‚1.4451456114649772 2loss/SRqqt4/assignment_loss‚1.2796120756616196 epoch‚7 % _timestamp‚1.7884248056228158e+09 0loss/FRt2/assignment_loss‚1.0264116298068653 1loss/FBt1/assignment_loss‚0.14292346148249635 2loss/SRqqt3/assignment_loss‚1.6095022718280048 1loss/FBt4/assignment_loss‚0.08149752034417664 1loss/SRqqt1/detection_loss‚2.3048765310219355 1loss/SRqqt4/detection_loss‚1.6855675851305325 .loss/FBt4/detection_loss‚2.399541755517324 /loss/FRt4/detection_loss‚2.2082806813716886 0loss/FRt4/assignment_loss‚2.4440269124507905 .loss/FRt1/detection_loss‚2.612559389609557 /loss/FRt2/detection_loss‚1.5476506077430465 /loss/FBt1/detection_loss‚1.9556898420507258 /loss/FBt2/detection_loss‚0.6730778270601472 /loss/FRt1/assignment_loss‚2.752850022071447 2loss/SRqqt2/assignment_loss‚0.9636621466396671 ,loss/FRt3/detection_loss‚2.6505180199941šüˆù"Ù _step‚700 0loss/FRt3/assignment_loss‚2.2849216808875403 2loss/SRqqt1/assignment_loss‚1.4451456114649772 2loss/SRqqt4/assignment_loss‚1.2796120756616196 epoch‚7 % _timestamp‚1.7884248056228158e+09 0loss/FRt2/assignment_loss‚1.0264116298068653 1loss/FBt1/assignment_loss‚0.14292346148249635 2loss/SRqqt3/assignment_loss‚1.6095022718280048 1loss/FBt4/assignment_loss‚0.08149752034417664 1loss/SRqqt1/detection_loss‚2.3048765310219355 1loss/SRqqt4/detection_loss‚1.6855675851305325 .loss/FBt4/detection_loss‚2.399541755517324 /loss/FRt4/detection_loss‚2.2082806813716886 0loss/FRt4/assignment_loss‚2.4440269124507905 .loss/FRt1/detection_loss‚2.612559389609557 /loss/FRt2/detection_loss‚1.5476506077430465 /loss/FBt1/detection_loss‚1.9556898420507258 /loss/FBt2/detection_loss‚0.6730778270601472 /loss/FRt1/assignment_lossy·–Úׂ2.752850022071447 2loss/SRqqt2/assignment_loss‚0.9636621466396671 ,loss/FRt3/detection_loss‚2.6505180199941 1loss/SRqqt3/detection_loss‚1.7235885752459703 %loss/total_loss‚37.95335398547823 trainer/global_step‚8499 _runtime‚8276.372617058 /loss/FBt2/assignment_loss‚0.059673810527691 0loss/FBt3/assignment_loss‚0.5505886146565899 1loss/SRqqt2/detection_loss‚1.1148502101239406 /loss/FBt3/detection_loss‚2.4363372027873993¼»Õi=›ú"‚(: éääÔÿÿžx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.066' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.813200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[»£mû"‚(:â éääÔàÈýz disk.rbd1.out‚ 4.04931640625 disk.sda2.in‚0) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17% disk.nvme0n1.out‚37491.0986328125 network.sent‚ 2.1145696e+07 disk.sda.in‚ 7.36328125 disk.rbd0.out‚0 disk.sda3.out‚ 401.54296875 proc.memory.rssMB‚ 2861.78125) disk./.usagePercent‚73.71846084461022 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 243.1953125 disk.rbd0.in‚0 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚37491.0986328125 disk.nvme0n1.in‚5.3125" disk.dm-0.out‚37491.0986328125 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.rbd4.in‚0 network.recv‚ 2.918943e+06 disk.sda.out‚ 401.58203125 disk.sda3.in‚7.3125 disk.rbd2.in‚0 disk.sda1.out‚0$ disk./.usageGB‚1098.458339691162 disk.rbd3.out‚0 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 401.54296875 disk.sda1.in‚ 0.05078125$ memory_percent‚96.90461158752441' proc.memory.availableMB‚ 507.1484375 disk.rbd3.in‚0 disk.dm-0.in‚5.3125 disk.rbd4.out‚0 cpu‚25.304407881630514@Q Ä›ü"‚(: øääÔŪz gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚109.032' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.806399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/øSÉmý"‚(:â øääÔÖ€¶| disk.dm-1.out‚ 402.1328125 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 402.171875$ memory_percent‚96.90349102020264 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 402.1328125 network.recv‚ 2.924394e+06 disk.sda2.in‚0 proc.memory.rssMB‚ 2861.78125 cpu‚25.30394019393835 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125) proc.memory.percent‚17.46692657470703 disk.dm-0.in‚5.3125# disk.dm-0.out‚37591.24462890625 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚37591.24462890625 disk.sda.in‚ 7.36328125& disk.nvme0n1.out‚37591.24462890625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 network.sent‚ 2.1253041e+07) disk./.usagePercent‚73.71847006088146 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 243.2109375% disk./.usageGB‚1098.4584770202637 disk.rbd1.out‚ 4.04931640625 disk.nvme0n1.in‚5.3125 disk.rbd4.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 507.33203125óýdþ"‚(:… ‡åäÔË ”x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ë†voÿ"‚(:ä ‡åäÔ凳z( disk.nvme0c0n1.out‚37690.28369140625 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125) proc.memory.percent‚17.46692657470703( disk./.usagePercent‚73.7184726209568 disk.rbd1.in‚0& disk.nvme0n1.out‚37690.28369140625 disk.sda2.in‚0 disk.dm-1.out‚ 402.46484375 disk.rbd2.in‚0$ memory_percent‚96.90351486206055 proc.cpu.threads‚17 network.recv‚ 2.929403e+06 disk.rbd1.out‚ 4.04931640625 proc.memory.rssMB‚ 2861.78125 cpu‚25.303466040247613 network.sent‚ 2.1259165e+07% disk./.usageGB‚1098.4585151672363 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0 disk.sda.out‚ 402.50390625 disk.sda3.out‚ 402.46484375 disk.dm-0.in‚5.3125& proc.memory.availableMB‚ 507.328125 disk.sda.in‚ 7.36328125# disk.dm-0.out‚37690.28369140625 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.rbd2.out‚ 243.2734375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0[Ï ÷2€#- _wandbruntime‚8320  _runtime‚8320ã‹Ý*Æ#À & lr-AdamW/pg1‚0.0001920160338702681 & lr-AdamW/pg2‚0.0001920160338702681 trainer/global_step‚8549 $ _timestamp‚1.788424850222144e+09 _runtime‚8320.972061621 _step‚701éDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150202131 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 _step‚702 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222711044 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92499644 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222711044 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚339334557084672 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15886987215360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15887215418880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚355221544300032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364105407191040 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚339334575087616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380268935737856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 380781753 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16163805756928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364105431131136 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222685374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚355221790506496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158096381 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321315285767168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16163528546816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 % _timestamp‚1.7884248502228363e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15886987215360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15840864213654 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222685374 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321315268727808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380807425 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57701812 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚337156373629970 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚337156132941462 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚355221544300032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 _runtime‚8320.973012753 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380269236888064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚355221790506496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222711044 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92499644 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222711044 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚339334557084672 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15886987215360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15887215418880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚355221544300032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364105407191040 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚339334575087616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380268935737856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 380781753 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16163805756928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364105431131136 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222685374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚355221790506496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158096381 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321315285767168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16163528546816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 % _timestamp‚1.7884248502228363e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15886987215360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15840864213654 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222685374 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321315268727808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380807425 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57701812 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚337156373629970 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚337156132941462 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚355221544300032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 _runtime‚8320.973012753 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380269236888064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚355221790506496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150202131 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 _step‚702¾c*Yp2†#- _wandbruntime‚8321  _runtime‚8321=ðÿ}«‡#ñM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364146576887296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15889053126656 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92509999 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15842920262610 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158114098 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚380311968125440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 % _timestamp‚1.7884248510813642e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 380825023 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380850721 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222710925 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 _step‚703 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158114098 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364146600827392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321351588225024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 380825023 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15842696613254 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16165391238144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321351571185664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150218585 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158114100 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92510001 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_end/inactsó˜^L ive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15889053126656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚355261719938560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380312269262336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158114100 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 _runtime‚8321.832107812 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222710925 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚355261966158336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57707925 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150217924 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚339372913031680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222736621 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339372895028736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚337194267798918 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 380850721 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339372895028736ïghü&ˆ#öM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15889053126656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚355261719938560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380312269262336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158114100 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 _runtime‚8321.832107812 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222710925 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚355261966158336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57707925 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150217924 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚339372913031680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222736621 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339372895028736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚337194267798918 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 380850721 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339372895028736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364146576887296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15889053126656 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92509999 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15842920262610 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158114098 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚380311968125440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 % _timestamp‚1.7884248510813642e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 380825023 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380850721 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222710925 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 _step‚703 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158114098 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364146600827392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321351588225024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 380825023 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15842696613254 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16165391238144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321351571185664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150218585 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158114100 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92510001 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¿††ÔË2‰#- _wandbruntime‚8321  _runtime‚8321¼ÝnYŠ#Ó /loss/FBt1/detection_loss‚2.4565553963184357 %loss/total_loss‚42.54675373688335 1loss/FBt3/assignment_loss‚0.03588101388387648 /loss/FRt1/detection_loss‚2.7546228215549933 .loss/FRt3/detection_loss‚2.433554182201624 2loss/SRqqt3/assignment_loss‚1.5400871766389024 trainer/global_step‚8549 0loss/FBt2/assignment_loss‚0.1371643861314432 0loss/SRqqt4/detection_loss‚1.893412188316385 % _timestamp‚1.7884248510835228e+09 1loss/FBt4/assignment_loss‚0.35525523083560984 1loss/SRqqt2/detection_loss‚1.0451267987253174 0loss/FRt2/assignment_loss‚1.6888654037397735 0loss/FBt1/assignment_loss‚0.5473492207362748 /loss/FRt2/detection_loss‚1.7295231344748516 .loss/FRt4/detection_loss‚2.466450210774972 .loss/FBt4/detection_loss‚2.482081986963749 _runtime‚8321.833378907 1loss/SRqqt1/assignment_loss‚1.721187428985205 1loss/SRqqt2/assignment_loss‚0.913933686158132 1loss/SRqqt1/detection_loss‚2.2872647038764424 /loss/FBt2/detection_loss‚0.5247985197794299 /loss/FBt3/detection_loss‚3.2092944298471724 _step‚704 /loss/FRt4/assignment_loss‚2.824656280604276 epoch‚7 0loss/FRt1/assignment_loss‚3.2050535263437214 0loss/FRt3/assignment_loss‚2.8588430136442184 2loss/SRqqt4/assignment_loss‚1.5029852059669793 0loss/SRqqt3/detection_loss‚1.932807790381568×åfô^‹#Ø epoch‚7 0loss/FRt1/assignment_loss‚3.2050535263437214 0loss/FRt3/assignment_loss‚2.8588430136442184 2loss/SRqqt4/assignment_loss‚1.5029852059669793 0loss/SRqqt3/detection_loss‚1.932807790381568 /loss/FBt1/detection_loss‚2.4565553963184357 %loss/total_loss‚42.54675373688335 1loss/FBt3/assignment_loss‚0.03588101388387648 /loss/FRt1/detection_loss‚2.7546228215549933 .loss/FRt3/detection_loss‚2.433554182201624 2loss/SRqqt3/assignment_loss‚1.5400871766389024 trainer/global_step‚8549 0loss/FBt2/assignment_loss‚0.1371643861314432 0loss/SRqqt4/detection_loss‚1.893412188316385 % _timestamp‚1.7884248510835228e+09 1loss/FBt4/assignment_loss‚0.35525523083560984 1loss/SRqqt2/detection_loss‚1.0451267987253174 0loss/FRt2/assignment_loss‚1.6888654037397735 0loss/FBt1/assignment_loss‚0.5473492207362748 /loss/FRt2/detection_loss‚1.7295231344748516 .loss/FRt4/detection_loss‚2.466450210774972 .loss/FBt4/detection_loss‚2.482081986963749 _runtime‚8321.833378907 1loss/SRqqt1/assignment_loss‚1.721187428985205 1loss/SRqqt2/assignment_loss‚0.913933686158132 1loss/SRqqt1/detection_loss‚2.2872647038764424 /loss/FBt2/detection_loss‚0.5247985197794299 /loss/FBt3/detection_loss‚3.2092944298471724 _step‚704 /loss/FRt4/assignment_loss‚2.824656280604276Às€ÕŒ#‚(:„ –åäÔ¶“›y gpu.0.gpu‚7.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ÖÞn#‚(:ã –åäÔçùÎz disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 cpu‚25.303009421983035 network.sent‚ 2.1264993e+07 disk.nvme0n1.in‚5.3125 disk.rbd0.in‚0 disk.rbd2.out‚ 244.86328125 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125# disk./.usageGB‚1098.45853805542 disk.dm-1.out‚ 403.08203125 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚37789.59228515625 disk.sda1.out‚0 proc.memory.rssMB‚ 2861.78125' disk./.usagePercent‚73.718474157002 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 403.08203125 disk.rbd1.in‚0 disk.rbd1.out‚ 4.04931640625 disk.dm-0.in‚5.3125 proc.cpu.threads‚17 network.recv‚ 2.933504e+06& disk.nvme0n1.out‚37789.59228515625# disk.dm-0.out‚37789.59228515625 disk.sda.out‚ 403.12109375$ memory_percent‚96.90794944763184' proc.memory.availableMB‚ 506.6015625) proc.memory.percent‚17.46692657470703 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625 Ô)¶Ž#‚(:… ¥åäÔñ½³x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.266' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°Ê:¿o#‚(:ä ¥åäÔµ”ÿz proc.memory.rssMB‚ 2861.78125 cpu‚25.302553450464888 disk.dm-1.out‚ 403.51953125 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda3.in‚7.3125# disk.dm-0.out‚37789.89697265625) proc.memory.percent‚17.46692657470703 disk.sda.in‚ 7.36328125 disk.rbd2.out‚ 244.92578125 disk.sda.out‚ 403.55859375 disk.rbd1.out‚ 4.19775390625 disk.sda2.in‚0) disk./.usagePercent‚73.71847466901707 disk.rbd2.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 proc.cpu.threads‚17 network.recv‚ 2.9411e+06 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda3.out‚ 403.51953125 disk.nvme0n1.in‚5.3125% disk./.usageGB‚1098.4585456848145 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0& disk.nvme0n1.out‚37789.89697265625 disk.sda2.out‚ 0.0390625& proc.memory.availableMB‚ 506.046875 disk.rbd0.out‚0 network.sent‚ 2.1373982e+07 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚37789.89697265625 disk.rbd3.in‚0$ memory_percent‚96.91133499145508Á8”#‚(:„ ´åäÔ£¾¬x gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.175' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åey•g‘#‚(:Ü ´åäÔï²ûz disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 network.recv‚ 2.944287e+06% disk./.usageGB‚1098.4585456848145 disk.sda3.in‚7.3125$ memory_percent‚96.91295623779297) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda2.out‚ 0.0390625 disk.sda3.out‚ 404.078125% disk.nvme0n1.out‚37889.7666015625 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.rbd2.out‚ 244.94140625 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 4.19775390625 disk.dm-0.in‚5.3125 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 404.1171875 cpu‚25.302101626197196) disk./.usagePercent‚73.71847466901707 disk.rbd0.out‚0% proc.memory.availableMB‚ 505.78125 proc.memory.rssMB‚ 2861.78125 disk.nvme0c0n1.in‚5.3125' disk.nvme0c0n1.out‚37889.7666015625 network.sent‚ 2.137925e+07" disk.dm-0.out‚37889.7666015625 disk.dm-1.out‚ 404.078125² ±2’#- _wandbruntime‚8366  _runtime‚8366s¨JÞÈ“# trainer/global_step‚8599 $ _timestamp‚1.788424895854801e+09 _runtime‚8366.604753432 _step‚705 ' lr-AdamW/pg1‚0.00018709611533996776 ' lr-AdamW/pg2‚0.00018709611533996776ï=ÍÍ”#Ç ' lr-AdamW/pg1‚0.00018709611533996776 ' lr-AdamW/pg2‚0.00018709611533996776 trainer/global_step‚8599 $ _timestamp‚1.788424895854801e+09 _runtime‚8366.604753432 _step‚705ÁåsÊ«2•#- _wandbruntime‚8366  _runtime‚8366bœHë'–#åO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15978886352384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15932721434010 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151042258 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚357230340640256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341251454287872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 382974277 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15978886352384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚323130408659968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 _runtime‚8366.605712289 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58024255 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚382421077010432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357230586846720 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158982329 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151041649 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚366163915943936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚366163892003840 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 223966276 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357230340640256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341251472290816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16257185006592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357230586846720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚339062889405470 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341251472290816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15979114555904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93017396 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 trainer/global_step‚8599 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223991946 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 382948605 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158982331 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158982329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚339063130093978 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16257453828096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341251454287872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 _step‚706 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚323130391620608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15979114555904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58024862 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223991946 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93017394 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15932497784862 # _timestamp‚1.78842489585557e+09¯E»‚—#êO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15978886352384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚323130408659968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 _runtime‚8366.605712289 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58024255 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚382421077 %]Žn#010432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357230586846720 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158982329 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151041649 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚366163915943936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚366163892003840 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 223966276 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357230340640256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341251472290816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16257185006592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357230586846720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚339062889405470 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341251472290816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15979114555904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93017396 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 trainer/global_step‚8599 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223991946 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 382948605 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158982331 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158982329 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚339063130093978 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16257453828096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341251454287872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 _step‚706 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚323130391620608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15979114555904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58024862 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223991946 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93017394 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15932497784862 # _timestamp‚1.78842489585557e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15978886352384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15932721434010 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151042258 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚357230340640256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341251454287872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 382974277 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624·ŒæÇ2˜#- _wandbruntime‚8367  _runtime‚8367´ T®÷&™#ñM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224017523 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚357270516278784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚357270762498560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151059487 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16259037179392 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151058848 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 383017573 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93027751 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341289810234880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341289792231936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 382991875 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224017523 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159000050 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _step‚707Š½¾ü&š#öM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚357270762498560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151059487 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16259037179392 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151058848 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 383017573 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93027751 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341289810234880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341289792231936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 382991875 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224017523 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159000050 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _step‚707 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224017523 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚357270516278784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800ÃÃû2›#- _wandbruntime‚8367  _runtime‚8367æ{=n]œ#× 2loss/SRqqt1/assignment_loss‚1.8120924337682398 1loss/FBt1/assignment_loss‚0.14332857984118164 1loss/SRqqt1/detection_loss‚2.0168576328591867 %loss/total_loss‚40.26143217719693 epoch‚7 _runtime‚8367.454651656 0loss/FRt3/assignment_loss‚2.7422227278351783 trainer/global_step‚8599 % _timestamp‚1.7884248967048335e+09 0loss/FRt1/assignment_loss‚3.1962760993412562 2loss/SRqqt2/assignment_loss‚0.8781822017763796 2loss/SRqqt3/assignment_loss‚1.4048071321698703 1loss/FBt3/assignment_loss‚0.12795193432248197 1loss/SRqqt2/detection_loss‚0.9119159015552962 0loss/FBt2/assignment_loss‚0.0792063209227653 /loss/FRt4/assignment_loss‚3.131631374359131 /loss/FRt1/detection_loss‚2.6709252091390745 /loss/FRt2/detection_loss‚1.3300640221818216 -loss/FRt4/detection_loss‚2.53930170520358 1loss/SRqqt3/detection_loss‚1.6615152063646488 .loss/FBt4/detection_loss‚2.805511736869812 1loss/FBt4/assignment_loss‚0.20120797434356064 0loss/FRt2/assignment_loss‚1.5840020818369729 .loss/FRt3/detection_loss‚2.531521587818861 1loss/SRqqt4/detection_loss‚2.0321964140360556 /loss/FBt3/detection_loss‚1.9535118838151295 _step‚708 2loss/SRqqt4/assignment_loss‚1.5480923246747504 /loss/FBt1/detection_loss‚2.3809251685937247 /loss/FBt2/detection_loss‚0.5781845235679712øíµb#Ü /loss/FBt2/detection_loss‚0.5781845235679712 2loss/SRqqt1/assignment_loss‚1.8120924337682398 1loss/FBt1/assignment_loss‚0.14332857984118164 1loss/SRqqt1/detection_loss‚2.0168576328591867 %loss/total_loss‚40.26143217719693 epoch‚7 _runtime‚8367.454651656 0loss/FRt3/assignment_loss‚2.7422227278351783 trainer/global_step‚8599 % _timestamp‚1.7884248967048335e+09 0loss/FRt1/assignment_loss‚3.1962760993412562 2loss/SRqqt2/assignment_loss‚0.8781822017763796 2loss/SRqqt3/assignment_loss‚1.4048071321698703 1loss/FBt3/assignment_loss‚0.12795193432248197 1loss/SRqqt2/detection_loss‚0.9119159015552962 0loss/FBt2/assignment_loss‚0.0792063209227653 /loss/FRt4/assignment_loss‚3.131631374359131 /loss/FRt1/detection_loss‚2.6709252091390745 /loss/FRt2/detection_loss‚1.3300640221818216 -loss/FRt4/detection_loss‚2.53930170520358 1loss/SRqqt3/detection_loss‚1.6615152063646488 .loss/FBt4/detection_loss‚2.805511736869812 1loss/FBt4/assignment_loss‚0.20120797434356064 0loss/FRt2/assignment_loss‚1.5840020818369729 .loss/FRt3/detection_loss‚2.531521587818861 1loss/SRqqt4/detection_loss‚2.0321964140360556 /loss/FBt3/detection_loss‚1.9535118838151295 _step‚708 2loss/SRqqt4/assignment_loss‚1.5480923246747504 /loss/FBt1/detection_loss‚2.3809251685937247Ä ãólž#‚(:… ÃåäÔ¹¨x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.962' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•Ó5D­Ÿ#‚(:Ô ÃåäÔù‡·{ disk.sda.out‚ 404.6640625 disk.rbd4.out‚0 disk.sda3.in‚7.3125 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125" disk.dm-0.out‚37988.7978515625' disk.nvme0c0n1.out‚37988.7978515625$ memory_percent‚96.91314697265625( disk./.usagePercent‚73.7184792771527% disk./.usageGB‚1098.4586143493652 disk.sda3.out‚404.625 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.31%šd²25) proc.memory.percent‚17.46692657470703 cpu‚25.30164841155599 disk.sda.in‚ 7.36328125% disk.nvme0n1.out‚37988.7978515625 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd1.in‚0 disk.rbd1.out‚ 4.22998046875 disk.sda2.in‚0 disk.dm-1.out‚404.625 disk.rbd2.in‚0" proc.memory.availableMB‚505.75 proc.memory.rssMB‚ 2861.78125 proc.cpu.threads‚17 disk.rbd4.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 network.recv‚ 2.94877e+06! disk.rbd2.out‚245.15673828125 disk.rbd3.in‚0 network.sent‚ 2.1385264e+07¦·" #‚(:„ ÒåäÔ–»¢x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.235' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.647 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ǤRvk¡#‚(:à ÒåäÔÿéžz$ disk./.usageGB‚1098.458595275879 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125# disk.nvme0n1.out‚38088.23828125 disk.sda.in‚ 7.36328125 cpu‚25.301207040686148% disk.nvme0c0n1.out‚38088.23828125 disk.rbd1.out‚ 4.22998046875 disk.dm-1.in‚7.3125! disk.rbd2.out‚245.17236328125( proc.memory.availableMB‚ 505.54296875 proc.cpu.threads‚17 network.recv‚ 2.955549e+06 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 405.69921875$ memory_percent‚96.91441059112549) proc.memory.percent‚17.46692657470703 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 405.73828125 disk.dm-1.out‚ 405.69921875 disk.rbd2.in‚0 proc.memory.rssMB‚ 2861.78125 network.sent‚ 2.149374e+07 disk.sda1.out‚0 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-0.out‚38088.23828125) disk./.usagePercent‚73.71847799711502 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.sda3.in‚7.3125o!|uŽ¢#‚(:ƒ áåäÔ’ª³x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.302 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þpo£#‚(:ä áåäÔøÁûz disk.rbd2.in‚0! disk.rbd2.out‚245.23486328125 disk.drbd1777.in‚ 1.03515625 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625( proc.memory.availableMB‚ 505.41796875 proc.memory.rssMB‚ 2861.78125 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 406.22265625 disk.dm-0.in‚5.3125 disk.dm-1.in‚7.3125 network.recv‚ 2.960163e+06 disk.nvme0c0n1.in‚5.3125% disk.nvme0n1.out‚38187.2958984375 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚0' disk.nvme0c0n1.out‚38187.2958984375 disk.sda3.in‚7.3125% disk./.usageGB‚1098.4586181640625 disk.sda.out‚ 406.26171875) disk./.usagePercent‚73.71847953316023 disk.rbd0.out‚0 disk.rbd3.in‚0" disk.dm-0.out‚38187.2958984375 cpu‚25.30075530427273 proc.cpu.threads‚17 network.sent‚ 2.1500162e+07 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚5.3125) proc.memory.percent‚17.46692657470703 disk.rbd1.out‚ 4.29296875 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 406.22265625$ memory_percent‚96.91517353057861Z53o2¤#- _wandbruntime‚8411  _runtime‚8411iÖŠ©È¥# ' lr-AdamW/pg2‚0.00018222080726390915 trainer/global_step‚8649 $ _timestamp‚1.788424941013286e+09 _runtime‚8411.763159139 _step‚709 ' lr-AdamW/pg1‚0.00018222080726390915F÷˜Í¦#Ç ' lr-AdamW/pg2‚0.00018222080726390915 trainer/global_step‚8649 $ _timestamp‚1.788424941013286e+09 _runtime‚8411.763159139 _step‚709 ' lr-AdamW/pg1‚0.00018222080726390915Å.Bª%2§#- _wandbruntime‚8411  _runtime‚8411-îŠì'¨#æO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚359239383186944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343168369494016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16070785489408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16350945044480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16024131356070 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93535144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 225247178 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚340969645869478 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225247178 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58347268 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚359239136980480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16350684611584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159868279 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚343168369494016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343168351491072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _runtime‚8411.764211371 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159868281 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343168351491072 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 trainer/global_step‚8649 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151882414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384573061428224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚368222400756736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159868279 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58346620 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324945514513408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚359239383186944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384573345801216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚340969886557986 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385115457 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚359239136980480 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225272848Z£)ñ'©#ëO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16070785489408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16350945044480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16024131356070 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93535144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 225247178 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚340969645869478 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225247178 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58347268 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚359239136980480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16350684611584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159868279 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚343168369494016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343168351491072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _runtime‚8411.764211371 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159868281 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343168351491072 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 trainer/global_step‚8649 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151882414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384573061428224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚368222400756736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159868279 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58346620 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324945514513408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚359239383186944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384573345801216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚340969886557986 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385115457 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚359239136980480 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225272848 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚359239383186944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343168369494016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428ÆܪFm2ª#- _wandbruntime‚8412  _runtime‚8412:’ô«#ñM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 385158727 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151899505 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151898873 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159885998 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16352820977152 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16026187405026 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159886000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚343206707438080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58354004 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93545499 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16072623183872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93545501 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 385158727 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16352554266112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚343206689435136 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225298425 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385184425 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16025963755670 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 _runtime‚8412.619495967 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚343206707438080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159885998 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚359279558838784 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159886000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16072623183872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368263570452992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚359279558838784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384616100779008 trainer/global_step‚8649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58353374 % _timestamp‚1.7884249418689382e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚341007780726934 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚343206689435136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384616391430144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368263546512896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚341008021415650 K=DeviceStatsMoÑ,Gnitor.on_train_batch_end/active.small_pool.freed‚ 225272729 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 _step‚711 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324981834010624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225298425 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 385184425 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ü1v÷ü&¬#öM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚343206689435136 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225298425 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385184425 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16025963755670 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 _runtime‚8412.619495967 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚343206707438080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159885998 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚359279558838784 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159886000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16072623183872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368263570452992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚359279558838784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384616100779008 trainer/global_step‚8649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58353374 % _timestamp‚1.7884249418689382e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚341007780726934 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚343206689435136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384616391430144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368263546512896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚341008021415650 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225272729 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 _step‚711 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324981834010624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225298425 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 385184425 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 385158727 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151899505 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151898873 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159885998 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16352820977152 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16026187405026 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159886000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚343206707438080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58354004 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93545499 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16072623183872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93545501 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 385158727 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16352554266112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299Ç'k,w2­#- _wandbruntime‚8412  _runtime‚8412žP­[®#Õ .loss/FBt1/detection_loss‚2.536702275276184 epoch‚7 1loss/FBt3/assignment_loss‚0.10487370588816702 1loss/SRqqt1/detection_loss‚2.3066094679137072 1loss/SRqqt4/detection_loss‚1.7806720318312341 trainer/global_step‚8649 % _timestamp‚1.7884249418707688e+09 1loss/SRqqt2/assignment_loss‚1.002356385661719 2loss/SRqqt3/assignment_loss‚1.3124978375785492 /loss/FRt2/detection_loss‚1.7048305882060009 /loss/FBt2/detection_loss‚0.5369752638359019 /loss/FRt3/assignment_loss‚2.501237180672194 /loss/FRt4/assignment_loss‚3.224481635767481 2loss/SRqqt1/assignment_loss‚2.0518552743985006 1loss/SRqqt4/assignment_loss‚1.808365872565736 1loss/FBt2/assignment_loss‚0.06264656586192492 1loss/SRqqt3/detection_loss‚1.7833895507980795 .loss/FBt3/detection_loss‚4.686102628707886 _runtime‚8412.620579498 /loss/FRt3/detection_loss‚2.6118808749474978 0loss/FRt1/assignment_loss‚3.6889781692753667 1loss/FBt1/assignment_loss‚0.16921571742487787 1loss/FBt4/assignment_loss‚0.35175796190095654 /loss/FRt1/detection_loss‚3.3696202259996664 1loss/SRqqt2/detection_loss‚0.9313019046130089 /loss/FRt2/assignment_loss‚1.567997113518093 /loss/FBt4/detection_loss‚2.6287625630696616 %loss/total_loss‚44.81458528087989 _step‚712 .loss/FRt4/detection_loss‚2.091474485167499S.àt`¯#Ú 1loss/SRqqt1/detection_loss‚2.3066094679137072 1loss/SRqqt4/detection_loss‚1.7806720318312341 trainer/global_step‚8649 % _timestamp‚1.7884249418707688e+09 1loss/SRqqt2/assignment_loss‚1.002356385661719 2loss/SRqqt3/assignment_loss‚1.3124978375785492 /loss/FRt2/detection_loss‚1.7048305882060009 /loss/FBt2/detection_loss‚0.5369752638359019 /loss/FRt3/assignment_loss‚2.501237180672194 /loss/FRt4/assignment_loss‚3.224481635767481 2loss/SRqqt1/assignment_loss‚2.0518552743985006 1loss/SRqqt4/assignment_loss‚1.808365872565736 1loss/FBt2/assignment_loss‚0.06264656586192492 1loss/SRqqt3/detection_loss‚1.7833895507980795 .loss/FBt3/detection_loss‚4.686102628707886 _runtime‚8412.620579498 /loss/FRt3/detection_loss‚2.6118808749474978 0loss/FRt1/assignment_loss‚3.6889781692753667 1loss/FBt1/assignment_loss‚0.16921571742487787 1loss/FBt4/assignment_loss‚0.35175796190095654 /loss/FRt1/detection_loss‚3.3696202259996664 1loss/SRqqt2/detection_loss‚0.9313019046130089 /loss/FRt2/assignment_loss‚1.567997113518093 /loss/FBt4/detection_loss‚2.6287625630696616 %loss/total_loss‚44.81458528087989 _step‚712 .loss/FRt4/detection_loss‚2.091474485167499 .loss/FBt1/detection_loss‚2.536702275276184 epoch‚7 1loss/FBt3/assignment_loss‚0.10487370588816702È¥Gá°#‚(:… ðåäÔ–…šx gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.936' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ¯ÜUq±#‚(:æ ðåäÔÕÑÈz disk.rbd1.out‚ 4.29296875 disk.nvme0n1.in‚5.3125! disk.rbd2.out‚246.82470703125$ memory_percent‚96.91381454467773 cpu‚25.300312217338025 network.recv‚ 2.963555e+06 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd0.in‚0% disk./.usageGB‚1098.4585990905762 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 proc.memory.rssMB‚ 2861.78125 disk.rbd4.out‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚38286.70361328125 disk.dm-0.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 406.74609375 disk.rbd2.in‚0) disk./.usagePercent‚73.71847825312256 disk.sda2.in‚0& proc.memory.availableMB‚ 505.640625 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.rbd1.in‚0 disk.rbd0.out‚0 network.sent‚ 2.1505426e+07( disk.nvme0c0n1.out‚38286.70361328125# disk.dm-0.out‚38286.70361328125 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 406.78515625 disk.sda3.out‚ 406.74609375 disk.dm-1.in‚7.3125) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd4.in‚0Ҝَ²#‚(:ƒ ÿåäÔ€«x gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$`ïi³#‚(:Þ ÿåäÔˆîÊ{ disk.sda.in‚ 7.36328125 proc.memory.rssMB‚ 2861.78125 network.sent‚ 2.1614301e+07 disk.rbd2.in‚0! disk.rbd2.out‚246.88720703125 disk.sda3.in‚7.3125 disk.sda2.out‚ 0.0390625 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚38286.99267578125 disk.rbd1.out‚ 4.35595703125% proc.memory.availableMB‚ 505.71875 cpu‚25.29986195984142 proc.cpu.threads‚17 network.recv‚ 2.97058e+06# disk.dm-0.out‚38286.99267578125 disk.dm-1.out‚ 407.2109375 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 407.2109375 disk.dm-0.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.rbd3.out‚0) disk./.usagePercent‚73.71848516532597 disk.sda.out‚407.25 disk.rbd1.in‚0$ memory_percent‚96.91333770751953% disk./.usageGB‚1098.4587020874023 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚38286.99267578125) proc.memory.percent‚17.46692657470703 disk.sda1.in‚ 0.05078125ê¬ïÚŽ´#‚(:ƒ ŽæäÔ¬á–x gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.43' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aæ.9tµ#‚(:é ŽæäÔ¥ÚÂz disk.dm-1.out‚ 408.43359375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚38387.49267578125$ memory_percent‚96.91500663757324 disk.rbd0.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚ 2861.78125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 408.47265625' proc.memory.availableMB‚ 505.4453125 cpu‚25.299426078756305 disk.rbd1.out‚ 4.35595703125 disk.dm-0.in‚5.3125 disk.nvme0c0n1.in‚5.3125 network.sent‚ 2.1620161e+07) disk./.usagePercent‚73.71846750080611$ disk./.usageGB‚1098.458438873291& disk.nvme0n1.out‚38387.49267578125# disk.dm-0.out‚38387.49267578125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.nvme0n1.in‚5.3125! disk.rbd2.out‚246.90283203125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 408.43359375) proc.memory.percent‚17.46692657470703 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 network.recv‚ 2.974644e+06 disk.sda2.in‚0ÏCø;2¶#- _wandbruntime‚8456  _runtime‚8456`ÛrÉ·#à ' lr-AdamW/pg1‚0.00017739112564898817 ' lr-AdamW/pg2‚0.00017739112564898817 trainer/global_step‚8699 % _timestamp‚1.7884249861901817e+09 _runtime‚8456.940119213 _step‚713ábøθ#È _step‚713 ' lr-AdamW/pg1‚0.00017739112564898817 ' lr-AdamW/pg2‚0.00017739112564898817 trainer/global_step‚8699 % _timestamp‚1.7884249861901817e+09 _runtime‚8456.940119213É*ïu€2¹#- _wandbruntime‚8456  _runtime‚8456…>ƒí'º#çO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚345085266697216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚342876402333486 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16115988576426 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚345085248694272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386725329744896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚326760637406208 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152725116 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚361247933320704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16444444175360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚345085266697216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16115764927278 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚345085248694272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387307981 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 trainer/global_step‚8699 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚361248179527168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94052896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386725034886144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16162912829952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 226553750 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚342876643021994 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152724473 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _step‚714 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16162912829952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 226553750 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94052894 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚641 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226528080 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚361247933320704 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 160754231 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160754229 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58671579 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚370280885569536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 160754231 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚361248179527168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226528080 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚326760637406208 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152725116 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚361247933320704 dQDevicH‡ra7eStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16444444175360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚345085266697216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16115764927278 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚345085248694272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387307981 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 trainer/global_step‚8699 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚361248179527168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94052896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386725034886144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16162912829952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 226553750 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚342876643021994 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152724473 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 _step‚714 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16162912829952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 226553750 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94052894 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚641 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226528080 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚361247933320704 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 160754231 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160754229 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58671579 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚370280885569536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 160754231 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚361248179527168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226528080 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚361288355179008 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226553631 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342914777879658 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387351277 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160771950 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚386768375123968 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370322031325696 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160771948 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚345123604641280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311622656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152741326 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16446032175616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16117597326878 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚8457.802423924 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94063249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386768063501312 trainer/global_step‚8699 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16164522320896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 _step‚715 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226553631 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160771948 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16164522320896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 226579327 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚345123586638336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58678763 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160771950 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16446319858176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16164750537728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884249870518444e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚361288355179008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370322055265792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚326796939864064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ï®!ü&¾#öM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚361288355179008 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226553631 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342914777879658 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387351277 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160771950 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚386768375123968 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370322031325696 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160771948 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚345123604641280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311622656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152741326 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16446032175616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16117597326878 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚8457.802423924 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94063249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386768063501312 trainer/global_step‚8699 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16164522320896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 _step‚715 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226553631 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160771948 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16164522320896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 226579327 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚345123586638336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58678763 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160771950 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16446319858176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16164750537728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884249870518444e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚361288355179008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370322055265792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚326796939864064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152742014 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚361288108959232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚345123604641280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚345123586638336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚342914537190942 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16117820976234 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16164750537728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94063251 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299Ë '2¿#- _wandbruntime‚8457  _runtime‚8457\œFm_À#Ù 2loss/SRqqt3/assignment_loss‚1.1695688201360546 1loss/FBt3/assignment_loss‚0.07643562805787951 /loss/FBt4/detection_loss‚2.3953787750667996 %loss/total_loss‚40.90975778117479 trainer/global_step‚8699 % _timestamp‚1.7884249870537627e+09 0loss/FRt2/assignment_loss‚1.3116015286549279 0loss/FBt1/assignment_loss‚0.5749206024657951 /loss/FRt2/detection_loss‚1.7421936437325634 /loss/FBt1/detection_loss‚2.6061335653066635 /loss/FBt2/detection_loss‚0.6364252914089519 /loss/FBt3/detection_loss‚2.9301684126257896 _runtime‚8457.803586854 /loss/FRt1/assignment_loss‚3.137222880666906 1loss/FBt2/assignment_loss‚0.11180157356329581 1loss/FBt4/assignment_loss‚0.10543158257173167 1loss/SRqqt1/detection_loss‚2.5560267584191427 1loss/SRqqt2/detection_loss‚0.8994241525468073 _step‚716 0loss/FRt4/assignment_loss‚3.3175694679512695 1loss/SRqqt3/detection_loss‚1.6846943002826762 epoch‚7 2loss/SRqqt2/assignment_loss‚0.7429697117768228 1loss/SRqqt4/detection_loss‚1.9488890522972067 2loss/SRqqt4/assignment_loss‚1.8067718113356448 /loss/FRt1/detection_loss‚2.4590599359097807 /loss/FRt3/detection_loss‚2.3970677204430104 /loss/FRt4/detection_loss‚2.5833950358297595 /loss/FRt3/assignment_loss‚2.157009260356426 1loss/SRqqt1/assignment_loss‚1.559598269768887gE­dÁ#Þ /loss/FRt1/detection_loss‚2.4590599359097807 /loss/FRt3/detection_loss‚2.3970677204430104 /loss/FRt4/detection_loss‚2.5833950358297595 /loss/FRt3/assignment_loss‚2.157009260356426 1loss/SRqqt1/assignment_loss‚1.559598269768887 2loss/SRqqt3/assignment_loss‚1.1695688201360546 1loss/FBt3/assignment_loss‚0.07643562805787951 /loss/FBt4/detection_loss‚2.3953787750667996 %loss/total_loss‚40.90975778117479 trainer/global_step‚8699 % _timestamp‚1.7884249870537627e+09 0loss/FRt2/assignment_loss‚1.3116015286549279 0loss/FBt1/assignment_loss‚0.5749206024657951 /loss/FRt2/detection_loss‚1.7421936437325634 /loss/FBt1/detection_loss‚2.6061335653066635 /loss/FBt2/detection_loss‚0.6364252914089519 /loss/FBt3/detection_loss‚2.9301684126257896 _runtime‚8457.803586854 /loss/FRt1/assignment_loss‚3.137222880666906 1loss/FBt2/assignment_loss‚0.11180157356329581 1loss/FBt4/assignment_loss‚0.10543158257173167 1loss/SRqqt1/detection_loss‚2.5560267584191427 1loss/SRqqt2/detection_loss‚0.8994241525468073 _step‚716 0loss/FRt4/assignment_loss‚3.3175694679512695 1loss/SRqqt3/detection_loss‚1.6846943002826762 epoch‚7 2loss/SRqqt2/assignment_loss‚0.7429697117768228 1loss/SRqqt4/detection_loss‚1.9488890522972067 2loss/SRqqt4/assignment_loss‚1.8067718113356448ÌÖÒ$ýÂ#‚(:… æäÔþªx gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.584' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pßÓ¬nÃ#‚(:ã æäÔÄã‡{ disk.rbd1.out‚ 4.35595703125 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 408.96875 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚38486.54736328125 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd2.out‚246.9931640625 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 338.17578125 disk.sda1.out‚0$ memory_percent‚97.93593883514404) proc.memory.percent‚17.46692657470703 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 409.0078125 disk.sda3.in‚7.3125# disk.dm-0.out‚38486.54736328125 network.sent‚ 2.1626445e+07) disk./.usagePercent‚73.71847492502461% disk./.usageGB‚1098.4585494995117 disk.dm-1.in‚7.3125 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 cpu‚25.298952408641554 network.recv‚ 2.979529e+06 disk.sda3.out‚ 408.96875 disk.rbd1.in‚0& disk.nvme0n1.out‚38486.54736328125 proc.memory.rssMB‚ 2861.78125 proc.cpu.threads‚17HP7ZÄ#‚(:… ¬æäÔçÿŠx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.999' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s6spÔÅ#‚(:á ¬æäÔ·´œz disk.sda3.in‚7.3125 disk.rbd1.out‚ 4.35595703125& disk.nvme0n1.out‚38586.52587890625 disk.dm-0.in‚5.3125% disk./.usageGB‚1098.4585304260254 disk.rbd0.in‚0 dic×pm˜sk.drbd1777.in‚ 1.03515625 disk.rbd1.in‚0' proc.memory.availableMB‚ 338.0859375 network.recv‚ 2.985607e+06 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚38586.52587890625 proc.cpu.threads‚17 disk.rbd2.out‚247.0048828125 disk.drbd1777.out‚0) disk./.usagePercent‚73.71847364498694 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 409.5859375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda2.out‚ 0.0390625) proc.memory.percent‚17.46692657470703 network.sent‚ 2.1734541e+07 disk.dm-1.out‚ 409.5859375 disk.sda1.in‚ 0.05078125 disk.sda.out‚409.625 proc.memory.rssMB‚ 2861.78125 cpu‚25.29848899414191 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚38586.52587890625 disk.sda1.out‚0$ memory_percent‚97.93648719787598x‚\ Æ#‚(:„ »æäÔå³–x gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.681' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µvÝ_rÇ#‚(:ç »æäÔÊôïz) disk./.usagePercent‚73.71853329474241 disk.rbd0.out‚0 disk.sda3.out‚ 410.01171875& disk.nvme0n1.out‚38685.59228515625# disk.dm-0.out‚38685.59228515625 disk.dm-1.out‚ 410.01171875 disk.rbd3.out‚0$ memory_percent‚97.93462753295898) proc.memory.percent‚17.46692657470703 cpu‚25.298047241989874 disk.rbd1.out‚ 4.35595703125 disk.dm-0.in‚5.3125 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 network.sent‚ 2.174055e+07 disk.sda3.in‚7.3125 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.rbd2.out‚248.6416015625 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 network.recv‚ 2.990168e+06 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125& proc.memory.availableMB‚ 338.390625 proc.memory.rssMB‚ 2861.78125% disk./.usageGB‚1098.4594192504883 disk.sda.out‚ 410.05078125 disk.sda2.in‚0( disk.nvme0c0n1.out‚38685.59228515625 proc.cpu.threads‚17‹ÎÝ2È#- _wandbruntime‚8502  _runtime‚8502Á @MÉÉ#à % _timestamp‚1.7884250314899979e+09 _runtime‚8502.239939714 _step‚717 ' lr-AdamW/pg1‚0.00017260807699361424 ' lr-AdamW/pg2‚0.00017260807699361424 trainer/global_step‚8749êÎÊ#È % _timestamp‚1.7884250314899979e+09 _runtime‚8502.239939714 _step‚717 ' lr-AdamW/pg1‚0.00017260807699361424 ' lr-AdamW/pg2‚0.00017260807699361424 trainer/global_step‚8749ÍküW—2Ë#- _wandbruntime‚8502  _runtime‚8502ÉT­Qì'Ì#æO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153567686 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161640179 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚328575777338368 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389449161 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚347002145897472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚328575760299008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚388877412241920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153567011 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚363256729660928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚388877115286016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚363256729660928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚344783158797494 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚363256975867392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94570644 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚372339370382336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16538041859584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚718 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16254811966976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58997040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16254811966976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161640181 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16537768843776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 227808982 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚675 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58996367 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚344783399486002 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16254583763456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚347002163900416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347002145897472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚347002163900416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 trainer/global_step‚8749 $ _timestamp‚1.788425031490745e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227834652 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚363256975867392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389474833 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227834652 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16207622147634 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262Yf»0ñ'Í#ëO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16538041859584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚718 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16254811966976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58997040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16254811966976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161640181 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16537768843776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 227808982 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚675 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58996367 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚344783399486002 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16254583763456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚347002163900416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347002145897472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚347002163900416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 trainer/global_step‚8749 $ _timestamp‚1.788425031490745e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227834652 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚363256975867392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389474833 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227834652 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16207622147634 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153567686 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161640179 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚328575777338368 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389449161 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚347002145897472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚328575760299008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚388877412241920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153567011 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚363256729660928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚388877115286016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚363256729660928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚344783158797494 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚363256975867392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94570644 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚372339370382336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561Î'nœZ2Î#- _wandbruntime‚8503  _runtime‚85031hÍ $Ï#ñM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16256421457920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389492431 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389518129 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 227860229 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347040501844480 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161657900 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚328612062756864 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _runtime‚8503.100684809 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153584487 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚372380540078592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347040501844480 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚344821293654950 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _step‚719 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚363297151519232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚363297151519232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16539908420608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347040483841536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161657898 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚372380516138496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389518129 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161657898 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94580999 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16539631223808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347040483841536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59002850 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388920147362304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16209454547442 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16256421457920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16209230898086 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153583849 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227860229 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 % _timestamp‚1.7884250323501184e+09 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceÇÕ\çStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8749 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227834533 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚328612079796224kÍü&Ð#öM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16256421457920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389492431 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389518129 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 227860229 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347040501844480 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161657900 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚328612062756864 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _runtime‚8503.100684809 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153584487 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚372380540078592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347040501844480 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚344821293654950 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _step‚719 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚363297151519232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚363297151519232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16539908420608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347040483841536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161657898 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚372380516138496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389518129 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161657898 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94580999 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16539631223808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347040483841536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59002850 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388920147362304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16209454547442 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16256421457920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16209230898086 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153583849 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227860229 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 % _timestamp‚1.7884250323501184e+09 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8749 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227834533 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚328612079796224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698Ï¿®y2Ñ#- _wandbruntime‚8503  _runtime‚8503,^Ò#Ø 0loss/FRt3/assignment_loss‚3.0597257536390554 /loss/FRt4/assignment_loss‚2.745211939513683 /loss/FRt4/detection_loss‚2.4532569706439973 2loss/FBt4/assignment_loss‚0.014813671861442603 1loss/SRqqt4/detection_loss‚1.8344471360055299 /loss/FBt1/detection_loss‚2.8255282163619997 0loss/FBt1/assignment_loss‚0.5710838101804256 1loss/SRqqt2/detection_loss‚1.1270393191746646 /loss/FBt2/detection_loss‚0.6199651128900725 0loss/FRt2/assignment_loss‚1.9294421596404834 2loss/SRqqt3/assignment_loss‚1.0172441181756797 .loss/FRt1/detection_loss‚2.569254485946713 % _timestamp‚1.7884250323519015e+09 2loss/SRqqt1/assignment_loss‚2.1809851333498953 1loss/FBt3/assignment_loss‚0.32911796239204705 1loss/SRqqt3/detection_loss‚1.7667355508573594 &loss/total_loss‚43.310140636705924 1loss/SRqqt4/assignment_loss‚1.533031844892181 1loss/FBt2/assignment_loss‚0.11928440441312713 2loss/SRqqt2/assignment_loss‚0.9289801380176458 .loss/FRt2/detection_loss‚2.087933300397335 .loss/FRt3/detection_loss‚2.432004602059074 .loss/FBt3/detection_loss‚2.255014431476593 /loss/FBt4/detection_loss‚2.9220785328320096 _step‚720 0loss/FRt1/assignment_loss‚3.3277387510646474 1loss/SRqqt1/detection_loss‚2.6602232909202574 epoch‚7 trainer/global_step‚8749 _runtime‚8503.101726331ZÈR!cÓ#Ý 2loss/SRqqt2/assignment_loss‚0.9289801380176458 .loss/FRt2/detection_loss‚2.087933300397335 .loss/FRt3/detection_loss‚2.432004602059074 .loss/FBt3/detection_loss‚2.255014431476593 /loss/FBt4/detection_loss‚2.9220785328320096 _step‚720 0loss/FRt1/assignment_loss‚3.3277387510646474 1loss/SRqqt1/detection_loss‚2.6602232909202574 epoch‚7 trainer/global_step‚8749 _runtime‚8503.101726331 0loss/FRt3/assignment_loss‚3.0597257536390554 /loss/FRt4/assignment_loss‚2.745211939513683 /loss/FRt4/detection_loss‚2.4532569706439973 2loss/FBt4/assignment_loss‚0.014813671861442603 1loss/SRqqt4/detection_loss‚1.8344471360055299 /loss/FBt1/detection_loss‚2.8255282163619997 0loss/FBt1/assignment_loss‚0.5710838101804256 1loss/SRqqt2/detection_loss‚1.1270393191746646 /loss/FBt2/detection_loss‚0.6199651128900725 0loss/FRt2/assignment_loss‚1.9294421596404834 2loss/SRqqt3/assignment_loss‚1.0172441181756797 .loss/FRt1/detection_loss‚2.569254485946713 % _timestamp‚1.7884250323519015e+09 2loss/SRqqt1/assignment_loss‚2.1809851333498953 1loss/FBt3/assignment_loss‚0.32911796239204705 1loss/SRqqt3/detection_loss‚1.7667355508573594 &loss/total_loss‚43.310140636705924 1loss/SRqqt4/assignment_loss‚1.533031844892181 1loss/FBt2/assignment_loss‚0.11928440441312713Ð˧<›Ô#‚(: ÊæäÔßÛx gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EÆoÕ#‚(:ä ÊæäÔ¸…Ãz disk.rbd1.out‚ 4.35595703125 disk.nvme0n1.in‚5.3125$ memory_percent‚97.93615341186523 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 410.6640625 disk.rbd4.out‚0 disk.sda.out‚ 410.703125 cpu‚25.297613314149245 disk.sda3.out‚ 410.6640625 disk.dm-0.in‚5.3125# disk.dm-0.out‚38785.13330078125 disk.rbd4.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.rbd1.in‚0 network.recv‚ 2.994448e+06) disk./.usagePercent‚73.71853329474241 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0& disk.nvme0n1.out‚38785.13330078125 proc.memory.rssMB‚ 2861.78125 network.sent‚ 2.1746506e+07) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 disk.drbd1777.out‚0% disk./.usageGB‚1098.4594192504883 disk.rbd2.out‚248.6572265625 disk.nvme0c0n1.in‚5.3125& proc.memory.availableMB‚ 338.140625 disk.sda2.in‚0 disk.rbd2.in‚0 disk.sda.in‚ 7.36328125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚38785.13330078125¼Ö#‚(:… ÙæäÔ¼å‘x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡rtl×#‚(:á ÙæäÔâ׶{ disk.sda1.out‚0& disk.nvme0n1.out‚38785.45751953125# disk.dm-0.out‚38785.45751953125 proc.memory.rssMB‚ 2861.78125 disk.dm-1.out‚ 410.9921875 disk.rbd2.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚38785.45751953125& proc.memory.availableMB‚ 338.109375 network.recv‚ 3.001935e+06 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.sda3.in‚7.3125 disk.rbd1.out‚ 4.35595703125 disk.nvme0n1.in‚5.3125 disk.drbd1777.out‚0 disk.sda.out‚ 411.03125 disk.dm-0.in‚5.3125 disk.rbd2.out‚248.7822265625$ memory_percent‚97.93634414672852% disk./.usageGB‚1098.4594192504883 disk.sda2.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0) proc.memory.percent‚17.46692657470703 network.sent‚ 2.185547e+07) disk./.usagePercent‚73.71853329474241 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 410.9921875 cpu‚25.29714396483936 proc.cpu.threads‚17í8ÒØ#‚(:‚ èæäÔ˜ŠŒx gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.68' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ¥û¢qÙ#‚(:æ èæäÔ–Õ{ disk.sda3.in‚7.3125 disk.sda2.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚38884.7626953125 disk.dm-1.out‚ 411.58203125 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.dm-0.in‚5.3125 disk.dm-1.in‚7.3125' disk.nvme0c0n1.out‚38884.7626953125 network.recv‚ 3.005436e+06% disk./.usageGB‚1098.4594192504883 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0 disk.nvme0n1.in‚5.3125 disk.rbd2.in‚0 disk.rbd0.out‚0$ memory_percent‚97.93636798858643 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 network.sent‚ 2.1860751e+07 disk.sda3.out‚ 411.58203125 disk.rbd1.in‚0 disk.drbd1777.out‚0 cpu‚25.29673871341948) disk./.usagePercent‚73.71853329474241 disk.sda.out‚ 411.62109375 disk.rbd1.out‚ 4.35595703125 disk.rbd2.out‚248.7978515625( proc.memory.availableMB‚ 338.10546875 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125" disk.dm-0.out‚38884.7626953125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125 proc.cpu.threads‚17 disk.rbd3.in‚0*êÅ­2Ú#- _wandbruntime‚8547  _runtime‚8547â>2ÉÛ#à _step‚721 ' lr-AdamW/pg1‚0.00016787265807795826 ' lr-AdamW/pg2‚0.00016787265807795826 trainer/global_step‚8799 % _timestamp‚1.7884250766110778e+09 _runtime‚8547.361019044$”λÎÜ#È % _timestamp‚1.7884250766110778e+09 _runtime‚8547.361019044 _step‚721 ' lr-AdamW/pg1‚0.00016787265807795826 ' lr-AdamW/pg2‚0.00016787265807795826 trainer/global_step‚8799ÑÑ+¯·2Ý#- _wandbruntime‚8547  _runtime‚8547§Àì'Þ#æO BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365265526001152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162526131 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16299255718842 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16346711104000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154411987 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 229115554 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95088394 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚346690155950010 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16299032069694 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229089884 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16346482900480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 229115554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59323591 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16346711104000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162526129 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95088396 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16631259872768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚330390900231168 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚365265772207616 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚348919043100672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59322919 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 _runtime‚ 8547.36202601 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162526131 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚346689915261502 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚365265526001152 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154411313 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348919061103616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚722 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0bb&vÈß#ëO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚365265772207616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚348919043100672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16631534985728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330390883191808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16346482900480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚374397855195136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365265526001152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162526131 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16299255718842 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16346711104000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154411987 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 229115554 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95088394 60DeviceStatsMonitor.on_train_batch_start/num_oomØm%)s‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚346690155950010 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16299032069694 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229089884 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16346482900480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 229115554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59323591 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16346711104000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162526129 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95088396 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16631259872768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚330390900231168 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚365265772207616 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚348919043100672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59322919 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 _runtime‚ 8547.36202601 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162526131 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚346689915261502 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚365265526001152 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154411313 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348919061103616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚722 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚348957381044736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 391659283 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348957399047680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59329875 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16633402775552 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229141131 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16348548811776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚391072427666944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162543850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59329194 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348957381044736 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154427943 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚346728290807674 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚346728050118958 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391684981 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287682560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 trainer/global_step‚8799 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229141131 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311622656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚374439024891392 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16301088118650 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚348957399047680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330427202689024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚365305947859456 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 391684981 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16348320594944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162543848 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ÏÓ=Sü&â#öM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229141131 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311622656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚374439024891392 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16301088118650 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚348957399047680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330427202689024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚365305947859456 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 391684981 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16348320594944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162543848 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚348957381044736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 391659283 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348957399047680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59329875 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16633402775552 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229141131 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16348548811776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚391072427666944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162543850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59329194 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348957381044736 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154427943 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚346728290807674 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚346728050118958 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391684981 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287682560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 trainer/global_step‚8799ÓQ§ë2ã#- _wandbruntime‚8548  _runtime‚8548³ËMG^ä#Ø /loss/FRt2/detection_loss‚1.7281088781143938 2loss/SRqqt2/assignment_loss‚1.0460787162676664 1loss/FBt1/assignment_loss‚0.34474218162682113 3loss/FBt3/assignment_loss‚0.0051627758990174955 1loss/SRqqt3/detection_loss‚1.7464013490825891 0loss/FBt2/detection_loss‚0.48131993695734493 2loss/SRqqt3/assignment_loss‚1.2851220854624574 1loss/FBt2/assignment_loss‚0.16071162972015626 .loss/FBt1/detection_loss‚2.133689363797506 trainer/global_step‚8799 # _timestamp‚1.78842507746703e+09 0loss/FRt1/assignment_loss‚3.5132974482351735 /loss/FRt4/detection_loss‚2.4402922166565846 1loss/SRqqt2/detection_loss‚1.0521645864411708 0loss/SRqqt4/detection_loss‚1.828724410317161 _step‚724 .loss/FRt3/detection_loss‚2.388523047281937 _runtime‚8548.216886998 0loss/FRt2/assignment_loss‚0.9381659128228013 0loss/SRqqt1/detection_loss‚2.658073874957421 %loss/total_loss‚40.21921165682117 epoch‚7 /loss/FRt3/assignment_loss‚2.658734791658141 /loss/FBt4/detection_loss‚2.5866917627198354 0loss/FRt4/assignment_loss‚3.1870088984326617 2loss/SRqqt4/assignment_loss‚1.1584536333314397 1loss/FBt4/assignment_loss‚0.11231924469133706 .loss/FBt3/detection_loss‚2.772648411137717 2loss/SRqqt1/assignment_loss‚1.1246519224608647 /loss/FRt1/detection_loss‚2.8681245787489797{Õyccå#Ý 1loss/FBt2/assignment_loss‚0.16071162972015626 .loss/FBt1/detection_loss‚2.133689363797506 trainer/global_step‚8799 # _timestamp‚1.78842507746703e+09 0loss/FRt1/assignment_loss‚3.5132974482351735 /loss/FRt4/detection_loss‚2.4402922166565846 1loss/SRqqt2/detection_loss‚1.0521645864411708 0loss/SRqqt4/detection_loss‚1.828724410317161 _step‚724 .loss/FRt3/detection_loss‚2.388523047281937 _runtime‚8548.216886998 0loss/FRt2/assignment_loss‚0.9381659128228013 0loss/SRqqt1/detection_loss‚2.658073874957421 %loss/total_loss‚40.21921165682117 epoch‚7 /loss/FRt3/assignment_loss‚2.658734791658141 /loss/FBt4/detection_loss‚2.5866917627198354 0loss/FRt4/assignment_loss‚3.1870088984326617 2loss/SRqqt4/assignment_loss‚1.1584536333314397 1loss/FBt4/assignment_loss‚0.11231924469133706 .loss/FBt3/detection_loss‚2.772648411137717 2loss/SRqqt1/assignment_loss‚1.1246519224608647 /loss/FRt1/detection_loss‚2.8681245787489797 /loss/FRt2/detection_loss‚1.7281088781143938 2loss/SRqqt2/assignment_loss‚1.0460787162676664 1loss/FBt1/assignment_loss‚0.34474218162682113 3loss/FBt3/assignment_loss‚0.0051627758990174955 1loss/SRqqt3/detection_loss‚1.7464013490825891 0loss/FBt2/detection_loss‚0.48131993695734493 2loss/SRqqt3/assignment_loss‚1.2851220854624574ÔaZæ#‚(:„ ÷æäÔ›Ú±x gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.289' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06‡Msç#‚(:è ÷æäÔŽ·÷z" disk.dm-0.out‚38983.7587890625 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd1.in‚0 cpu‚25.296282423950785 network.sent‚ 2.1866743e+07% disk.nvme0n1.out‚38983.7587890625 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 411.97265625 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125) proc.memory.percent‚17.46692657470703 disk.rbd3.out‚0( proc.memory.availableMB‚ 337.86328125% disk./.usageGB‚1098.4594192504883 disk.sda.in‚ 7.36328125 network.recv‚ 3.009726e+06 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda3.out‚ 411.97265625 disk.rbd1.out‚ 4.35595703125) disk./.usagePercent‚73.71853329474241 disk.dm-0.in‚5.3125! disk.rbd2.out‚248.88818359375 disk.sda.out‚ 412.01171875$ memory_percent‚97.93784618377686 proc.memory.rssMB‚ 2861.78125 proc.cpu.threads‚17' disk.nvme0c0n1.out‚38983.7587890625 disk.sda3.in‚7.3125?[ ƒè#‚(:… †çäÔ¨ƒŠx gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.687' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1374 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìl\}ké#‚(:à †çäÔîÙ¿{ disk.dm-0.in‚5.3125 network.sent‚ 2.1975297e+07% disk./.usageGB‚1098.4593963623047 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda.in‚ 7.36328125) disk./.usagePercent‚73.71853175869721 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.rbd1.in‚0) proc.memory.percent‚17.46692657470703 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 412.5234375% disk.nvme0n1.out‚39083.1142578125 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.dm-1.out‚ 412.5234375 proc.cpu.threads‚17 disk.nvme0n1.in‚5.3125" disk.dm-0.out‚39083.1142578125 network.recv‚ 3.016994e+06 disk.sda1.out‚0 disk.rbd1.out‚ 4.35595703125& proc.memory.availableMB‚ 337.859375 disk.sda1.in‚ 0.05078125 disk.sda.out‚412.5625 disk.dm-1.in‚7.3125$ memory_percent‚97.93787002563477 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 2861.78125 cpu‚25.295849038414303! disk.rbd2.out‚248.90380859375' disk.nvme0c0n1.out‚39083.1142578125$#Šnê#‚(:„ •çäÔÿÖŸx gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.854' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# ‘ÐQ!gpu.0.uncorrectedMemoryErrors‚0[k–mhë#‚(:Ý •çäÔ¡“âz disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 413.34375 disk.sda1.in‚ 0.05078125 disk.rbd0.out‚0 disk.rbd1.in‚0" disk.dm-0.out‚39182.1611328125$ memory_percent‚97.93658256530762 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 4.35595703125 disk.dm-1.in‚7.3125 disk.drbd1777.out‚0 disk.rbd4.out‚0' proc.memory.availableMB‚ 338.0703125$ disk./.usageGB‚1098.459617614746' disk.nvme0c0n1.out‚39182.1611328125 disk.rbd3.out‚0! disk.rbd2.out‚250.54052734375 disk.drbd1777.in‚ 1.03515625) proc.memory.percent‚17.46692657470703 cpu‚25.29542496080474 disk.sda3.in‚7.3125 network.sent‚ 2.1981633e+07( disk./.usagePercent‚73.7185466071342 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 413.3828125 proc.memory.rssMB‚ 2861.78125 network.recv‚ 3.021695e+06 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd4.in‚0% disk.nvme0n1.out‚39182.1611328125 disk.sda2.in‚0 disk.dm-1.out‚ 413.34375Š²Ù 2ì#- _wandbruntime‚8592  _runtime‚8592oµÖ\Çí#Á & lr-AdamW/pg1‚0.0001631858557562246 & lr-AdamW/pg2‚0.0001631858557562246 trainer/global_step‚8849 % _timestamp‚1.7884251218592498e+09 _runtime‚8592.609182119 _step‚725ïÎÌî#Æ % _timestamp‚1.7884251218592498e+09 _runtime‚8592.609182119 _step‚725 & lr-AdamW/pg1‚0.0001631858557562246 & lr-AdamW/pg2‚0.0001631858557562246 trainer/global_step‚8849ÕþÅ@F2ï#- _wandbruntime‚8592  _runtime‚8592¸£Eí'ð#çO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 230370786 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155253259 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16724760527872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚8849 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚367274322341376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332206006084608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16438610241024 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348596912414018 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16390889290050 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163412081 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95606144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚367274568547840 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚332206023123968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚367274568547840 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163412079 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 _step‚726 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393181367260160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚350835958306816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16438382037504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚350835940303872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163412081 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16725027252224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393782865 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155253916 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59647115 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 230396456 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230370786 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16438610241024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16390665640902 % _timestamp‚1.7884251218599753e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376456340007936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95606146 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚350835940303872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376456316067840 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59647770 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚348596671725510 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚367274322341376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 393808537 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16438382037504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393181076595712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚350835958306816Ýu'ò'ñ#ìO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16438382037504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393181076595712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚350835958306816 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 230370786 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155253259 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16724760527872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚8849 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚367274322341376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332206006084608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16438610241024 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348596912414018 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16390889290050 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163412081 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95606144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚367274568547840 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚332206023123968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚367274568547840 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163412079 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 _step‚726 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393181367260160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚350835958306816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16438382037504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚350835940303872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163412081 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16725027252224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393782865 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155253916 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59647115 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 230396456 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230370786 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16438610241024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16390665640902 % _timestamp‚1.7884251218599753e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376456340007936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95606146 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚350835940303872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376456316067840 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59647770 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚348596671725510 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚367274322341376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 393808537ÖÑæÓ²2ò#- _wandbruntime‚8593  _runtime‚8593ŸGc¬÷&ó#ñM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16440447948800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16392498040502 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163429798 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16392721689858 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376497509704192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚350874296250880 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16440219731968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 230396337 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚367314744199680 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 230422033 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393851833 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163429800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95616499 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332242308542464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 trainer/global_step‚8849 _step‚727 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚367314497979904 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16726903996928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16440219731968 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚350874296250880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 230422033 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16440447948800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163429800 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16726618411520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155269930 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚393224413701120 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348635047271682 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚350874278247936 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163429798 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59654093 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 % _timestamp‚1.7884251227324514e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 230396337 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚367314744199680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332242325581824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262²E>h+ô#öM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚367314744199680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332242325581824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 >8fÀÑ$DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16440447948800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16392498040502 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163429798 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16392721689858 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376497509704192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚350874296250880 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16440219731968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 230396337 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚367314744199680 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 230422033 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393851833 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163429800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95616499 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332242308542464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 trainer/global_step‚8849 _step‚727 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚367314497979904 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16726903996928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16440219731968 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚350874296250880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 230422033 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16440447948800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163429800 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16726618411520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155269930 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚393224413701120 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348635047271682 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚350874278247936 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163429798 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59654093 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 % _timestamp‚1.7884251227324514e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 230396337×*'¹¨2õ#- _wandbruntime‚8593  _runtime‚8593š(?n_ö#Ù _runtime‚8593.483838464 2loss/SRqqt2/assignment_loss‚0.9434485936439351 /loss/FRt2/detection_loss‚1.4727728977028667 1loss/SRqqt2/detection_loss‚0.6898366006973543 /loss/FRt1/assignment_loss‚2.948187175818852 /loss/FBt1/detection_loss‚2.1125054597854613 .loss/FBt4/detection_loss‚2.777312469482422 0loss/FRt4/assignment_loss‚3.0751610414849386 0loss/FBt2/assignment_loss‚0.1352581986840993 1loss/FBt4/assignment_loss‚0.15958781476510922 /loss/FRt3/detection_loss‚2.4955192384394733 1loss/SRqqt3/detection_loss‚1.8282095694317009 /loss/FBt3/detection_loss‚2.3395172506570816 &loss/total_loss‚39.880056894988286 epoch‚7 0loss/FRt2/assignment_loss‚1.4415915813939324 2loss/SRqqt4/assignment_loss‚1.4472831162481994 1loss/FBt1/assignment_loss‚0.06090451979016507 /loss/FRt1/detection_loss‚2.7849582088845115 /loss/FRt4/detection_loss‚2.5666475908623982 0loss/SRqqt1/detection_loss‚2.428063197992742 1loss/SRqqt4/detection_loss‚1.8586972012358196 0loss/FRt3/assignment_loss‚2.6541098329153927 1loss/SRqqt3/assignment_loss‚1.399245319079678 1loss/FBt3/assignment_loss‚0.22390290022303816 _step‚728 /loss/FBt2/detection_loss‚0.5664842762414691 trainer/global_step‚8849 % _timestamp‚1.7884251227340236e+09 2loss/SRqqt1/assignment_loss‚1.4708528395276517î¸IFd÷#Þ 2loss/SRqqt2/assignment_loss‚0.9434485936439351 /loss/FRt2/detection_loss‚1.4727728977028667 1loss/SRqqt2/detection_loss‚0.6898366006973543 /loss/FRt1/assignment_loss‚2.948187175818852 /loss/FBt1/detection_loss‚2.1125054597854613 .loss/FBt4/detection_loss‚2.777312469482422 0loss/FRt4/assignment_loss‚3.0751610414849386 0loss/FBt2/assignment_loss‚0.1352581986840993 1loss/FBt4/assignment_loss‚0.15958781476510922 /loss/FRt3/detection_loss‚2.4955192384394733 1loss/SRqqt3/detection_loss‚1.8282095694317009 /loss/FBt3/detection_loss‚2.3395172506570816 &loss/total_loss‚39.880056894988286 epoch‚7 0loss/FRt2/assignment_loss‚1.4415915813939324 2loss/SRqqt4/assignment_loss‚1.4472831162481994 1loss/FBt1/assignment_loss‚0.06090451979016507 /loss/FRt1/detection_loss‚2.7849582088845115 /loss/FRt4/detection_loss‚2.5666475908623982 0loss/SRqqt1/detection_loss‚2.428063197992742 1loss/SRqqt4/detection_loss‚1.8586972012358196 0loss/FRt3/assignment_loss‚2.6541098329153927 1loss/SRqqt3/assignment_loss‚1.399245319079678 1loss/FBt3/assignment_loss‚0.22390290022303816 _step‚728 /loss/FBt2/detection_loss‚0.5664842762414691 trainer/global_step‚8849 % _timestamp‚1.7884251227340236e+09 2loss/SRqqt1/assignment_loss‚1.4708528395276517 _runtime‚8593.483838464ØgûWø#‚(:… ¤çäÔù‰çx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ`ê où#‚(:ä ¤çäÔðŒîz% disk./.usageGB‚1098.4628829956055 disk.dm-0.in‚5.3125 disk.rbd2.in‚0 disk.rbd0.in‚0( disk./.usagePercent‚73.7187657495835 disk.dm-1.out‚ 414.0390625! disk.rbd2.out‚250.55615234375& proc.memory.availableMB‚ 339.296875 proc.memory.rssMB‚ 2861.78125# disk.dm-0.out‚39285.59326171875 disk.rbd1.out‚ 4.35595703125 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 414.078125 disk.sda3.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda3.out‚ 414.0390625 cpu‚25.294997371021594 proc.cpu.threads‚17 network.recv‚ 3.025367e+06 disk.sda2.out‚ 0.0390625 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0) proc.memory.percent‚17.46692657470703 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚5.3125 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚39285.59326171875 disk.sda1.in‚ 0.05078125& disk.nvme0n1.out‚39285.59326171875$ memory_percent‚97.92909622192383 network.sent‚ 2.1986999e+07s¦‹ ›ú#‚(: ³çäÔźßx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚100.902' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.180400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êf9pû#‚(:å ³çäÔÂÑüz disk.sda.out‚ 414.328125 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0( proc.memory.availableMB‚ 339.39453125) disk./.usagePercent‚73.70131474000449 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚5.3125 disk.rbd3.out‚0# disk.dm-0.out‚39285.86669921875 proc.memory.rssMB‚ 2861.78125 network.sent‚ 2.2095719e+07 network.recv‚ 3.032698e+06 disk.rbd0.out‚0( disk.nvme0c0n1.out‚39285.86669921875 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda2.in‚0 disk.rbd2.out‚ 250.646484375$ memory_percent‚97.92850017547607 cpu‚25.294542219997478 disk.sda3.out‚ 414.2890625 disk.rbd1.in‚0 disk.rbd1.out‚ 4.35595703125 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 414.2890625 disk.nvme0c0n1.in‚5.3125 disk.sda3.in‚7.3125& disk.nvme0n1.out‚39285.86669921875% disk./.usageGB‚1098.2028503417969 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚7.3125) proc.memory.percent‚17.46692657470703 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.363281258WljŽü#‚(:ƒ ÂçäÔ×Ûžx gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.37' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó¶Úÿký#‚(:à ÂçäÔö˜¶z) disk./.usagePercent‚73.64512467034422 disk.rbd4.in‚0 proc.memory.rssMB‚ 2861.78125 network.recv‚ 3.036745e+06 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 414.9296875& disk.nvme0n1.out‚39551.51806640625$ disk./.usageGB‚1097.365577697754 disk.sda3.in‚7.3125 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625 disk.dm-0.in‚5.3125( disk.nvme0c0n1.out‚39551.51806640625 disk.sda.in‚ 7.36328125 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 414.9296875 disk.rbd0.in‚0 cpu‚25.29412161087799 disk.sda1.out‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 250.646484375) proc.memory.percent‚17.46692657470703 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.rbd1.out‚ 4.35595703125 disk.rbd0.out‚0$ memory_percent‚97.93081283569336 disk.drbd1777.out‚0 disk.sda.out‚ 414.96875# disk.dm-0.out‚39551.51806640625& proc.memory.availableMB‚ 339.015625 disk.rbd3.out‚0 network.sent‚ 2.2101602e+07(Œ€æ2þ#- _wandbruntime‚8638  _runtime‚8638ÚÉåsÉÿ#à ' lr-AdamW/pg1‚0.00015854864675099255 ' lr-AdamW/pg2‚0.00015854864675099255 trainer/global_step‚8899 % _timestamp‚1.7884251672841802e+09 _runtime‚8638.034106128 _step‚729û÷!×΀$È trainer/global_step‚8899 % _timestamp‚1.7884251672841802e+09 _runtime‚8638.034106128 _step‚729 ' lr-AdamW/pg1‚0.00015854864675099255 ' lr-AdamW/pg2‚0.00015854864675099255Ù(¿Å2$- _wandbruntime‚8638  _runtime‚8638å¢+®í'‚$çO _runtime‚8638.034995854 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59970153 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352752855510016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352752837507072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚378514824820736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚369283118681600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352752855510016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231651688 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164298031 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16530281174528 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156094686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚378514800880640 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16530509378048 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395949717 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16530281174528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚395333288324096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚369283118681600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16482299212110 % _timestamp‚1.7884251672849052e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96123896ùÄoÆ߃$ìO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395975389 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚334021128977408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96123894 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚369283364888064 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 231677358 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16482522861258 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚395332997659648 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164298029 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164298031 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 trainer/global_step‚8899 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚352752837507072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16530509378048 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395949717 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16530281174528 E;DeviceStatsMonitor.on_train_batch_star°ª‡t/allocation.all.peak‚31397 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚395333288324096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚369283118681600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16482299212110 % _timestamp‚1.7884251672849052e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96123896 _runtime‚8638.034995854 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59970153 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352752855510016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352752837507072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚378514824820736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚369283118681600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352752855510016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231651688 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164298031 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16530281174528 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156094686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚378514800880640 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚350541803735690 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395992987 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16532118868992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96134249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 396018685 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 231677239 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352791193454080 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16820332997120 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16484355261066 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 231702935 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96134251 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16532347085824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164315750 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164315748 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚378555970576896 _runtime‚8638.901704276 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164315750 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156111226 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16484131611710 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚369323540539904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59976977 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378555994516992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352791175451136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 396018685 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164315748 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334057431435264 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚369323540539904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231702935 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231677239 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16820057897472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚350541563046974 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824¾/¾^ü&†$öM EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚369323540539904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231702935 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231677239 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16820057897472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚350541563046974 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚350541803735690 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395992987 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16532118868992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96134249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 396018685 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 231677239 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352791193454080 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16820332997120 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16484355261066 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 231702935 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96134251 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16532347085824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164315750 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164315748 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚378555970576896 _runtime‚8638.901704276 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164315750 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156111226 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16484131611710 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚369323540539904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59976977 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378555994516992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352791175451136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 396018685 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164315748 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334057431435264 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ÛÀQ÷2‡$- _wandbruntime‚8638  _runtime‚86386O¬^ˆ$Ø 0loss/FBt4/assignment_loss‚0.8092558311969487 /loss/FRt2/detection_loss‚1.7476398018730241 0loss/SRqqt1/detection_loss‚2.148995465523488 /loss/FBt3/detection_loss‚2.6863724112510683 2loss/SRqqt2/assignment_loss‚1.0148563229052696 1loss/FBt2/assignment_loss‚0.17222125628115886 .loss/FBt1/detection_loss‚2.580255953165201 2loss/SRqqt1/assignment_loss‚1.4488136587110725 0loss/FBt3/assignment_loss‚0.3975712171988562 /loss/FBt4/detection_loss‚2.1378793166233945 epoch‚7 trainer/global_step‚8899 % _timestamp‚1.7884251681529062e+09 /loss/FRt1/detection_loss‚2.3190225961994617 1loss/SRqqt2/detection_loss‚0.9871845635569702 1loss/SRqqt4/detection_loss‚1.7921459490762037 /loss/FBt2/detection_loss‚0.5998854896345411 _step‚732 %loss/total_loss‚40.23525382017723 0loss/FRt3/assignment_loss‚3.0149444001061574 0loss/FRt4/assignment_loss‚3.0057216157083926 2loss/SRqqt3/assignment_loss‚1.3836453448523556 /loss/FRt4/detection_loss‚2.2210167685280946 1loss/SRqqt3/detection_loss‚1.5436321638132398 0loss/FRt1/assignment_loss‚3.0240564219495085 2loss/SRqqt4/assignment_loss‚1.3022183558227969 0loss/FBt1/assignment_loss‚0.2625823995955531 /loss/FRt3/detection_loss‚2.6394054761954715 _runtime‚8638.902742481 0loss/FRt2/assignment_loss‚0.9959310404089994NèJc‰$Ý /loss/FRt2/detection_loss‚1.7476398018730241 0loss/SRqqt1/detection_loss‚2.148995465523488 /loss/FBt3/detection_loss‚2.6863724112510683 2loss/SRqqt2/assignment_loss‚1.0148563229052696 1loss/FBt2/assignment_loss‚0.17222125628115886 .loss/FBt1/detection_loss‚2.580255953165201 2loss/SRqqt1/assignment_loss‚1.4488136587110725 0loss/FBt3/assignment_loss‚0.3975712171988562 /loss/FBt4/detection_loss‚2.1378793166233945 epoch‚7 trainer/global_step‚8899 % _timestamp‚1.7884251681529062e+09 /loss/FRt1/detection_loss‚2.3190225961994617 1loss/SRqqt2/detection_loss‚0.9871845635569702 1loss/SRqqt4/detection_loss‚1.7921459490762037 /loss/FBt2/detection_loss‚0.5998854896345411 _step‚732 %loss/total_loss‚40.23525382017723 0loss/FRt3/assignment_loss‚3.0149444001061574 0loss/FRt4/assignment_loss‚3.0057216157083926 2loss/SRqqt3/assignment_loss‚1.3836453448523556 /loss/FRt4/detection_loss‚2.2210167685280946 1loss/SRqqt3/detection_loss‚1.5436321638132398 0loss/FRt1/assignment_loss‚3.0240564219495085 2loss/SRqqt4/assignment_loss‚1.3022183558227969 0loss/FBt1/assignment_loss‚0.2625823995955531 /loss/FRt3/detection_loss‚2.6394054761954715 _runtime‚8638.902742481 0loss/FRt2/assignment_loss‚0.9959310404089994 0loss/FBt4/assignment_loss‚0.8092558311969487ÜðÆ0—Š$‚(:… ÑçäÔÒ­¤x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 6¿ºs‹$‚(:è ÑçäÔÔ‹Óz# disk.dm-0.out‚39551.79931640625 disk.dm-1.out‚ 415.42578125 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125$ memory_percent‚97.93050289154053 proc.memory.rssMB‚ 2861.78125 disk.sda2.out‚ 0.0390625 cpu‚25.293697696951643 proc.cpu.threads‚17 disk.sda.in‚ 7.36328125 disk.sda.out‚ 415.46484375 disk.dm-1.in‚7.3125 disk.rbd2.out‚ 250.787109375 network.sent‚ 2.2107905e+07$ disk./.usageGB‚1097.365577697754 disk.rbd1.out‚ 4.35595703125 disk.drbd1777.in‚ 1.03515625) disk./.usagePercent‚73.64512467034422 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0& disk.nvme0n1.out‚39551.79931640625 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.rbd2.in‚0( proc.memory.availableMB‚ 339.06640625) proc.memory.percent‚17.46692657470703 network.recv‚ 3.041406e+06 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 415.42578125 disk.nvme0n1.in‚5.3125 disk.rbd0.in‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚39551.79931640625ÀŸ)ðŒ$‚(:… àçäÔžï·x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×^-X$‚(:Í àçäÔâß®z network.sent‚ 2.2215713e+07) disk./.usagePercent‚73.64512467034422 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125$ disk.nvme0c0n1.out‚ 39552.3671875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 cpu‚25.293278360285903" disk.nvme0n1.out‚ 39552.3671875 disk.drbd1777.in‚ 1.03515625 disk.sda3.in‚7.3125 network.recv‚ 3.046863e+06 disk.dm-1.out‚ 416.3828125 disk.sda1.in‚ 0.05078125) proc.memory.percent‚17.46692657470703 disk.dm-0.out‚ 39552.3671875 disk.rbd4.in‚0 proc.memory.availableMB‚339$ disk./.usageGB‚1097.365577697754 disk.dm-0.in‚5.3125 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 416.421875 disk.sda3.out‚ 416.3828125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd2.in‚0" memory_percent‚97.930908203125 proc.memory.rssMB‚ 2861.78125 disk.rbd1.out‚ 4.35595703125 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 disk.rbd2.out‚ 252.361328125 disk.rbd4.out‚0óÛ¤„šŽ$‚(: ïçäÔ ¡x gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.36' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.272000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒV$‚(:Ë ïçäÔ§¥Èz disk.dm-0.out‚ 39552.640625$ memory_percent‚97.92938232421875" proc.memory.availableMB‚339.25 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚5.3125! disk.nvme0n1.out‚ 39552.640625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.sda3.out‚ 416.8359375 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0$ disk./.usageGB‚1097.365577697754 disk.rbd4.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 416.8359375 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625# disk.nvme0c0n1.out‚ 39552.640625 disk.rbd1.out‚ 4.35595703125 disk.rbd2.out‚ 252.501953125 proc.memory.rssMB‚ 2861.78125 cpu‚25.29285753426273 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚416.875 disk.dm-1.in‚7.3125) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 network.recv‚ 3.051606e+06 disk.sda3.in‚7.3125 disk.sda2.out‚ 0.0390625 network.sent‚ 2.2221086e+07) disk./.usagePercent‚73.64512467034422 disk.drbd1777.out‚0è\šŸ2$- _wandbruntime‚8683  _runtime‚86839Œ²_‘$Á & lr-AdamW/pg1‚0.0001539619974496681 & lr-AdamW/pg2‚0.0001539619974496681 train\ÝÔher/global_step‚8949 % _timestamp‚1.7884252124601047e+09 _runtime‚8683.210035164 _step‚733~QL¹Ì’$Æ _step‚733 & lr-AdamW/pg1‚0.0001539619974496681 & lr-AdamW/pg2‚0.0001539619974496681 trainer/global_step‚8949 % _timestamp‚1.7884252124601047e+09 _runtime‚8683.210035164Ýœ+Õ2“$- _wandbruntime‚8683  _runtime‚8683[8Øí'”$çO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165183979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚380573285693440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚352410184653526 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380573309633536 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚354669734710272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚397485263344640 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16911953711104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96641644 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371291915021824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60297840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 232932590 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 trainer/global_step‚8949 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156939486 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16622180311552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16574156432466 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚397484964291584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335836251870208 _step‚734 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16622408515072 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 398142241 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 232958260 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96641646 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚352410425342034 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚354669752713216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16622408515072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299}þ%tò'•$ìO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 398116569 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚354669734710272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚352410184653526 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380573309633536 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚354669734710272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚397485263344640 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16911953711104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96641644 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371291915021824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60297840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 232932590 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 trainer/global_step‚8949 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156939486 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16622180311552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16574156432466 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚397484964291584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335836251870208 _step‚734 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16622408515072 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 398142241 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 232958260 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96641646 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚352410425342034 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚354669752713216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16622408515072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165183979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚380573285693440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398185537 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60304292 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚352448560199698 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚735 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚397528000775168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚335872571367424 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398159839 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96652001 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 _runtime‚8684.068117467 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 232983837 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16624018006016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60303657 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚335872554328064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156955656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380614479329792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380614455389696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚397528297717760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 398185537 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚371332336880128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚354708072654336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 398159839 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96651999 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 232958141 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165201700 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884252133173974e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚354708072654336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 232958141 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16575988832274 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165201700 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165201698 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16575765182918 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚371332336880128 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0E]ÇÞZ˜$öM E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚371332336880128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚354708072654336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pooB°Ã¢l.peak‚31168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 398159839 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96651999 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 232958141 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165201700 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 % _timestamp‚1.7884252133173974e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚354708072654336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 232958141 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16575988832274 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165201700 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165201698 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16575765182918 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚371332336880128 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8949 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚371332090660352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398185537 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60304292 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚352448560199698 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚735 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚397528000775168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚335872571367424 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398159839 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96652001 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 _runtime‚8684.068117467 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 232983837 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16624018006016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60303657 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚335872554328064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156955656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380614479329792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380614455389696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚397528297717760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 398185537 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673ßÕ-Î2™$- _wandbruntime‚8684  _runtime‚8684·h°;]š$× 1loss/SRqqt2/detection_loss‚1.0589135118893214 epoch‚7 _step‚736 /loss/FRt1/detection_loss‚2.7019854767454996 _runtime‚8684.069145733 1loss/SRqqt3/assignment_loss‚1.330797320918033 1loss/FBt2/assignment_loss‚0.11243071572014701 /loss/FBt2/detection_loss‚0.5745528722757453 &loss/total_loss‚40.491167154825476 % _timestamp‚1.7884252133193302e+09 0loss/FRt3/assignment_loss‚2.4022685027704007 1loss/FBt4/assignment_loss‚0.12057678084238432 /loss/FRt4/detection_loss‚2.5226245927524817 1loss/SRqqt3/detection_loss‚1.7884066669564498 /loss/FRt1/assignment_loss‚3.073280586136712 /loss/FRt3/detection_loss‚2.4764797222323534 .loss/FBt1/detection_loss‚2.108837154778567 /loss/FRt2/detection_loss‚1.6446429784099261 1loss/SRqqt1/detection_loss‚2.0808040343579792 2loss/SRqqt1/assignment_loss‚1.6606596282550268 trainer/global_step‚8949 1loss/SRqqt4/detection_loss‚2.4056427384677685 .loss/FBt3/detection_loss‚2.349155241792852 .loss/FBt4/detection_loss‚2.934928946197033 2loss/SRqqt2/assignment_loss‚0.8941492599834289 1loss/SRqqt4/assignment_loss‚1.420954796632654 1loss/FBt1/assignment_loss‚0.03447941509975036 1loss/FBt3/assignment_loss‚0.10299709483727136 /loss/FRt2/assignment_loss‚1.220943918493059 0loss/FRt4/assignment_loss‚3.4706551982806277X! &b›$Ü &loss/total_loss‚40.491167154825476 % _timestamp‚1.7884252133193302e+09 0loss/FRt3/assignment_loss‚2.4022685027704007 1loss/FBt4/assignment_loss‚0.12057678084238432 /loss/FRt4/detection_loss‚2.5226245927524817 1loss/SRqqt3/detection_loss‚1.7884066669564498 /loss/FRt1/assignment_loss‚3.073280586136712 /loss/FRt3/detection_loss‚2.4764797222323534 .loss/FBt1/detection_loss‚2.108837154778567 /loss/FRt2/detection_loss‚1.6446429784099261 1loss/SRqqt1/detection_loss‚2.0808040343579792 2loss/SRqqt1/assignment_loss‚1.6606596282550268 trainer/global_step‚8949 1loss/SRqqt4/detection_loss‚2.4056427384677685 .loss/FBt3/detection_loss‚2.349155241792852 .loss/FBt4/detection_loss‚2.934928946197033 2loss/SRqqt2/assignment_loss‚0.8941492599834289 1loss/SRqqt4/assignment_loss‚1.420954796632654 1loss/FBt1/assignment_loss‚0.03447941509975036 1loss/FBt3/assignment_loss‚0.10299709483727136 /loss/FRt2/assignment_loss‚1.220943918493059 0loss/FRt4/assignment_loss‚3.4706551982806277 1loss/SRqqt2/detection_loss‚1.0589135118893214 epoch‚7 _step‚736 /loss/FRt1/detection_loss‚2.7019854767454996 _runtime‚8684.069145733 1loss/SRqqt3/assignment_loss‚1.330797320918033 1loss/FBt2/assignment_loss‚0.11243071572014701 /loss/FBt2/detection_loss‚0.5745528722757453à ¬œ$‚(:‚ þçäÔ¬Öx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.33' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 L j$‚(:ß þçäÔóþ²z network.sent‚ 2.2226992e+07 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 417.4140625 disk.drbd1777.out‚0 disk.dm-1.in‚7.3125 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0 disk.nvme0n1.in‚5.3125 network.recv‚ 3.05612e+06 disk.dm-0.in‚5.3125" disk.dm-0.out‚39553.3603515625 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚39553.3603515625 disk.sda.in‚ 7.36328125( proc.memory.availableMB‚ 339.01953125 cpu‚25.292411679287863 disk.rbd0.out‚0 proc.memory.rssMB‚ 2861.78125$ disk./.usageGB‚1097.365608215332 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 417.4140625 disk.rbd1.out‚ 4.35595703125$ memory_percent‚97.93078899383545 disk.rbd2.out‚ 252.501953125( disk./.usagePercent‚73.6451267184045 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 417.453125) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625% disk.nvme0n1.out‚39553.3603515625©¨íšž$‚(: èäÔ£±„x gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.08' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.215999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nê°wkŸ$‚(:à èäÔß÷»z& disk.nvme0n1.out‚39553.78466796875 disk.dm-0.in‚5.3125 disk.sda2.in‚0 proc.memory.rssMB‚ 2861.78125 cpu‚25.292023227472054 network.sent‚ 2.2336029e+07( disk.nvme0c0n1.out‚39553.78466796875 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 418.0859375! disk.rbd2.out‚252.67041015625 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0& proc.memory.availableMB‚ 339.109375) proc.memory.percent‚17.46692657470703 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚39553.78466796875 disk.rbd4.out‚0 proc.cpu.threads‚17 disk.sda3.in‚7.3125 disk.dm-1.out‚ 418.0859375 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.drbd1777.out‚0( disk./.usagePercent‚73.6451267184045$ disk./.usageGB‚1097.365608215332 disk.rbd3.in‚0$ memory_percent‚97.93024063110352 network.recv‚ 3.063649e+06 disk.sda.out‚418.125 disk.rbd1.in‚0 disk.rbd1.out‚ 4.35595703125‰¦ÐË› $‚(: œèäԛפx gpu.0.gpu‚21.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.962' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.592399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Oäue¡$‚(:Ú œèäÔÓ–ëz disk.nvme0n1.in‚5.3125( disk.nvme0c0n1.out‚39554.45947265625' proc.memory.availableMB‚ 339.3515625) proc.memory.percent‚17.46692657470703& disk.nvme0n1.out‚39554.45947265625 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125 proc.memory.rssMB‚ 2861.78125 network.recv‚ 3.067008e+06 disk.drbd1777.in‚ 1.03515625$ disk./.usageGB‚1097.365623474121 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 418.5390625) disk./.usagePercent‚73.64512774243462 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda3.out‚418.5 disk.rbd1.out‚ 4.35595703125 cpu‚25.291577093861385 proc.cpu.threads‚17 disk.dm-1.out‚418.5 disk.rbd4.in‚0 disk.dm-0.in‚5.3125# disk.dm-0.out‚39554.45947265625 disk.rbd2.in‚0! disk.rbd2.out‚252.67822265625 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 2.2341333e+07 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.rbd1.in‚0$ memory_percent‚97.92876243591309ÏÊy 2¢$- _wandbruntime‚8728  _runtime‚8728™f§‡É£$à ' lr-AdamW/pg1‚0.00014942686370309123 ' lr-AdamW/pg2‚0.00014942686370309123 trainer/global_step‚8999 % _timestamp‚1.7884252577055216e+09 _runtime‚8728.455460635 _step‚737Ž”Œ™Î¤$È trainer/global_step‚8999 % _timestamp‚1.7884252577055216e+09 _runtime‚8728.455460635 _step‚737 ' lr-AdamW/pg1‚0.00014942686370309123 ' lr-AdamW/pg2‚0.00014942686370309123á4 º2¥$- _wandbruntime‚8728  _runtime‚8728wj®Yí'¦$çO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 trainer/global_step‚8999 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚738 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚8728.456443375 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 234239162 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚373300711362048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 234213492 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚373300711362048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399636938237440 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚356586631913472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400309093 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚354317181806042 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97159394 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚356586631913472 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16714079448576 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 234239162 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60619258 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 234213492 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16714079448576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97159396 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400283421 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16714307652096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 157778018 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166069931 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17005167731200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16665790003674 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166069929 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16665566354526 % _timestamp‚1.7884252577063475e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166069929 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚354316941117534 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚373300957568512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60618624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399637239387648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17005444941312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚373300957568512 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157778654 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚337651374763008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166069931 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16714307652096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚382631770506240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚356586649916416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚382631794446336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚337651391802368à[Ù§$ìO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16714079448576 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 234239162 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60619258 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 234213492 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16714079448576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97159396 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400283421 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16714307652096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 157778018 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166069931 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17005167731200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16665790003674 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166069929 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16665566354526 % _timestamp‚1.7884252577063475e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166069929 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚354316941117534 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚373300957568512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60618624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399637239387648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17005444941312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚373300957568512 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157778654 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚337651374763008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166069931 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16714307652096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚382631770506240 @8DeviceStatsMonitor.on_train_ñÈTNÙbatch_start/num_device_alloc‚561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚356586649916416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚382631794446336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚337651391802368 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 trainer/global_step‚8999 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚738 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚8728.456443375 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 234239162 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚373300711362048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 234213492 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚373300711362048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399636938237440 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚356586631913472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400309093 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚354317181806042 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97159394 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚356586631913472 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399679989408256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚356624969857536 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16715917143040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17007326402560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400326691 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚8729.324545615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 _step‚739 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 157795614 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚354355075974990 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚356624987860480 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 234239043 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 234239043 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788425258574076e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400352389 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 234264739 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97169751 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚373341133220352 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚8999 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166087648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚337687694260224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚654 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16715917143040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60625863 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157794960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚354355316663706 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16667622403482 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97169749Š³'û&ª$õM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157794960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚354355316663706 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16667622403482 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97169749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400326691 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚382672940202496 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚373341133220352 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399679989408256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚356624969857536 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16715917143040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17007326402560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400326691 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚8729.324545615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 _step‚739 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 157795614 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚354355075974990 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚356624987860480 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 234239043 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 234239043 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788425258574076e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400352389 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 234264739 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97169751 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚373341133220352 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚8999 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166087648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚337687694260224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚654 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16715917143040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60625863 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624ユœ2«$- _wandbruntime‚8729  _runtime‚8729ù”X¬$Ò 0loss/FRt3/assignment_loss‚2.3272407398774075 1loss/SRqqt2/assignment_loss‚0.928943576036545 2loss/SRqqt3/assignment_loss‚1.2293442135714754 1loss/SRqqt1/detection_loss‚2.1837227378572734 1loss/SRqqt2/detection_loss‚0.9063480214972515 /loss/FBt4/detection_loss‚2.2562237761237403 % _timestamp‚1.7884252585757728e+09 0loss/FRt1/assignment_loss‚2.9130409955978394 .loss/FRt4/detection_loss‚2.355137119976512 /loss/FBt1/detection_loss‚2.6809344461985996 &loss/total_loss‚39.190702439361985 _runtime‚ 8729.3256098 0loss/FBt3/assignment_loss‚0.2985460118215997 _step‚740 /loss/FRt2/assignment_loss‚1.198251104780606 1loss/SRqqt1/assignment_loss‚1.469393427670002 1loss/FBt1/assignment_loss‚0.12842428121283384 1loss/FBt2/assignment_loss‚0.13742104202203748 1loss/SRqqt3/detection_loss‚1.9047608500232502 /loss/FBt3/detection_loss‚2.2034914447711063 1loss/SRqqt4/assignment_loss‚1.646267432719469 .loss/FRt1/detection_loss‚2.608173870557063 -loss/FRt2/detection_loss‚1.63644956241894 0loss/FRt4/assignment_loss‚3.1428643505681646 1loss/FBt4/assignment_loss‚0.26124450226780027 1loss/SRqqt4/detection_loss‚1.9770018413979957 /loss/FBt2/detection_loss‚0.5256373444786189 trainer/global_step‚8999 .loss/FRt3/detection_loss‚2.271839745915853 epoch‚7OÒ]­$× 0loss/FRt3/assignment_loss‚2.3272407398774075 1loss/SRqqt2/assignment_loss‚0.928943576036545 2loss/SRqqt3/assignment_loss‚1.2293442135714754 1loss/SRqqt1/detection_loss‚2.1837227378572734 1loss/SRqqt2/detection_loss‚0.9063480214972515 /loss/FBt4/detection_loss‚2.2562237761237403 % _timestamp‚1.7884252585757728e+09 0loss/FRt1/assignment_loss‚2.9130409955978394 .loss/FRt4/detection_loss‚2.355137119976512 /loss/FBt1/detection_loss‚2.6809344461985996 &loss/total_loss‚39.190702439361985 _runtime‚ 8729.3256098 0loss/FBt3/assignment_loss‚0.2985460118215997 _step‚740 /loss/FRt2/assignment_loss‚1.198251104780606 1loss/SRqqt1/assignment_loss‚1.469393427670002 1loss/FBt1/assignment_loss‚0.12842428121283384 1loss/FBt2/assignment_loss‚0.13742104202203748 1loss/SRqqt3/detection_loss‚1.9047608500232502 /loss/FBt3/detection_loss‚2.2034914447711063 1loss/SRqqt4/assignment_loss‚1.646267432719469 .loss/FRt1/detection_loss‚2.608173870557063 -loss/FRt2/detection_loss‚1.63644956241894 0loss/FRt4/assignment_loss‚3.1428643505681646 1loss/FBt4/assignment_loss‚0.26124450226780027 1loss/SRqqt4/detection_loss‚1.9770018413979957 /loss/FBt2/detection_loss‚0.5256373444786189 trainer/global_step‚8999 .loss/FRt3/detection_loss‚2.271839745915853 epoch‚7äžT‚®$‚(:… «èäÔÌÁ¾x gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑãV³q¯$‚(:æ «èäÔ¼×{# disk.dm-0.out‚39556.56884765625 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.rbd1.out‚ 4.35595703125 proc.memory.rssMB‚ 2861.78125% disk./.usageGB‚1097.1948165893555! disk.rbd2.out‚252.81494140625 disk.sda1.out‚0 disk.sda.out‚ 418.99609375# memory_percent‚97.9301929473877' proc.memory.availableMB‚ 339.1171875 cpu‚25.29116218847263 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚39556.56884765625 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125) proc.memory.percent‚17.46692657470703 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd4.in‚0( disk.nvme0c0n1.out‚39556.56884765625 network.recv‚ 3.0713e+06 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 418.95703125 disk.rbd0.in‚0) disk./.usagePercent‚73.63366474908149 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 disk.sda3.out‚ 418.95703125 network.sent‚ 2.2347352e+07 disk.dm-0.in‚5.3125C¹7š°$‚(: ºèäÔ‘™ôy gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.751' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.750200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Uzßh±$‚(:Ý ºèäÔƒö|% disk./.usageGB‚1097.1948738098145 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda.out‚ 419.6171875! disk.rbd2.out‚254.38916015625 disk.rbd0.out‚0 disk.sda3.out‚ 419.578125) proc.memory.percent‚17.46692657470703 disk.dm-1.in‚7.3125# disk.dm-0.out‚39557.76904296875 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚39557.76904296875 disk.rbd0.in‚0 disk.sda3.in‚7.3125# proc.memory.availableMB‚339.375 network.sent‚ 2.2456029e+07 network.recv‚ 3.078119e+06 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0 disk.sda.in‚ 7.36328125 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚39557.76904296875 disk.rbd3.in‚0 proc.memory.rssMB‚ 2861.78125 cpu‚25.290743148976 proc.cpu.threads‚17 disk.dm-1.out‚ 419.578125 disk.rbd1.out‚ 4.35595703125( disk./.usagePercent‚73.6336685891945 disk.dm-0.in‚5.3125 disk.rbd4.in‚0 disk.rbd4.out‚0$ memory_percent‚97.92861938476562êXä²$‚(:… ÉèäÔ×ã«x gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 /œZj³$‚(:ß ÉèäÔØŠÏ{ disk.sda2.in‚0 disk.sda1.out‚0 disk.sda3.out‚420.125 disk.rbd1.out‚ 4.35595703125 disk.dm-0.in‚5.3125 disk.rbd3.out‚0 network.recv‚ 3.083101e+06& disk.nvme0n1.out‚39691.17919921875 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125( proc.memory.availableMB‚ 338.86328125 disk.nvme0n1.in‚5.3125 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.rbd3.in‚0) disk./.usagePercent‚73.64235236475592 disk.rbd1.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚254.48291015625 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚39691.17919921875 disk.sda.out‚ 420.1640625 proc.memory.rssMB‚ 2861.78125 network.sent‚ 2.2462364e+07 disk.dm-1.out‚420.125 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125$ memory_percent‚97.93174266815186 cpu‚25.29030677964911% disk./.usageGB‚1097.3242683410645# disk.dm-0.out‚39691.17919921875 disk.rbd0.out‚0 disk.sda3.in‚7.3125) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 disk.sda.in‚ 7.36328125ÊÉø2´$- _wandbruntime‚8773  _runtime‚8773cò”cɵ$à ' lr-AdamW/pg2‚0.00014494419062633708 trainer/global_step‚9049 % _timestamp‚1.7884253032114372e+09 _runtime‚8773.961384972 _step‚741 ' lr-AdamW/pg1‚0.00014494419062633708¨EW?ζ$È ' lr-AdamW/pg2‚0.00014494419062633708 trainer/global_step‚9049 % _timestamp‚1.7884253032114372e+09 _runtime‚8773.961384972 _step‚741 ' lr-AdamW/pg1‚0.00014494419062633708å¾¾™²2·$- _wandbruntime‚8773  _runtime‚8773Sš#¤§¸$çO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166955879 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17098972598272 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚375309507702272 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166955881 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402475945 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚401789251857408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚375309753908736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60940100 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166955881 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 235520064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384690279259136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16805978585600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358503547119616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16805978585600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17098699582464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚9049 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16757423574882 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚339466497655808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚401788954901504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235520064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚647 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16757199925734 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚8773.962258126 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16806206789120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 158617246 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97677144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚375309507702272 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166955881 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402475945 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚401789251857408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚375309753908736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60940100 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166955881 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 235520064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384690279259136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16805978585600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358503547119616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16805978585600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17098699582464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚9049 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16757423574882 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚339466497655808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚401788954901504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235520064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚647 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16757199925734 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚8773.962258126 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16806206789120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 158617246 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97677144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158633789 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚375349929560576 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358541885063680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 235519945 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17100559472128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚401832287721472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚9049 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚375349929560576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60946290 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚8774.844536368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60946912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16759255974690 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358541867060736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402519241 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166973600 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166973598 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235545641 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚339502817153024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 _step‚743 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235519945 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚358541867060736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97687499 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356262073127714 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚339502800113664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402519241 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚358541885063680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 235545641 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16807816280064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚401831984487424 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚384731448955392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16808044496896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158633789 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚375349929560576 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358541885063680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 235519945 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17100559472128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚401832287721472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚9049 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚375349929560576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60946290 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚8774.844536368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60946912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16759255974690 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358541867060736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402519241 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166973600 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166973598 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235545641 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚339502817153024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 _step‚743 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235519945 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚358541867060736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97687499 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356262073127714 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚339502800113664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402519241 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚358541885063680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 235545641 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16807816280064ç÷¸Ÿ©2½$- _wandbruntime‚8774  _runtime‚8774,.úÚ\¾$Ö 0loss/FRt3/assignment_loss‚3.1152051952150135 0loss/FRt4/assignment_loss‚2.9232986584687843 .loss/FRt3/detection_loss‚2.430606479114956 .loss/FBt1/detection_loss‚2.926500984600612 .loss/FBt4/detection_loss‚2.658981521924337 _runtime‚8774.845690381 epoch‚7 2loss/SRqqt2/assignment_loss‚0.8916892994730006 2loss/SRqqt3/assignment_loss‚1.6642669621011559 /loss/FRt2/detection_loss‚1.8078480138487758 1loss/SRqqt2/detection_loss‚1.0141733264026698 /loss/FRt1/assignment_loss‚2.647186067971316 1loss/FBt1/assignment_loss‚0.15772261001256993 1loss/FBt2/assignment_loss‚0.11391851099937096 1loss/FBt3/assignment_loss‚0.14744262301246636 1loss/SRqqt4/detection_loss‚1.8632813534982826 /loss/FBt2/detection_loss‚0.5351579840123457 0loss/FRt2/assignment_loss‚1.5600807936418624 2loss/SRqqt1/assignment_loss‚1.2588654508193333 1loss/FBt4/assignment_loss‚0.11438315925382388 $loss/total_loss‚41.7280129478233 1loss/SRqqt3/detection_loss‚2.0211152732372284 _step‚744 /loss/FBt3/detection_loss‚2.6077621057629585 trainer/global_step‚9049 $ _timestamp‚1.788425304095842e+09 2loss/SRqqt4/assignment_loss‚1.4459428111820116 /loss/FRt1/detection_loss‚2.7750501485936567 .loss/FRt4/detection_loss‚2.341428950889707 1loss/SRqqt1/detection_loss‚2.7061046637870647ù³a¿$Û $ _timestamp‚1.788425304095842e+09 2loss/SRqqt4/assignment_loss‚1.4459428111820116 /loss/FRt1/detection_loss‚2.7750501485936567 .loss/FRt4/detection_loss‚2.341428950889707 1loss/SRqqt1/detection_loss‚2.7061046637870647 0loss/FRt3/assignment_loss‚3.1152051952150135 0loss/FRt4/assignment_loss‚2.9232986584687843 .loss/FRt3/detection_loss‚2.430606479114956 .loss/FBt1/detection_loss‚2.926500984600612 .loss/FBt4/detection_loss‚2.658981521924337 _runtime‚8774.845690381 epoch‚7 2loss/SRqqt2/assignment_loss‚0.8916892994730006 2loss/SRqqt3/assignment_loss‚1.6642669621011559 /loss/FRt2/detection_loss‚1.8078480138487758 1loss/SRqqt2/detection_loss‚1.0141733264026698 /loss/FRt1/assignment_loss‚2.647186067971316 1loss/FBt1/assignment_loss‚0.15772261001256993 1loss/FBt2/assignment_loss‚0.11391851099937096 1loss/FBt3/assignment_loss‚0.14744262301246636 1loss/SRqqt4/detection_loss‚1.8632813534982826 /loss/FBt2/detection_loss‚0.5351579840123457 0loss/FRt2/assignment_loss‚1.5600807936418624 2loss/SRqqt1/assignment_loss‚1.2588654508193333 1loss/FBt4/assignment_loss‚0.11438315925382388 $loss/total_loss‚41.7280129478233 1loss/SRqqt3/detection_loss‚2.0211152732372284 _step‚744 /loss/FBt3/detection_loss‚2.6077621057629585 trainer/global_step‚9049èÃZšÀ$‚(: ØèäÔε—x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.496' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.899199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚH ÀsÁ$‚(:è ØèäÔͨœ{ network.sent‚ 2.2467635e+07 disk.rbd0.in‚0 disk.sda.out‚ 420.81640625$ memory_percent‚97.93295860290527 disk.rbd1.out‚ 4.35595703125# disk.dm-0.out‚39825.41064453125! disk.rbd2.out‚254.48291015625 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda3.out‚ 420.77734375 disk.rbd3.out‚0$ disk./.usageGB‚1097.455307006836 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 420.77734375 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚5.3125 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125' proc.memory.availableMB‚ 338.6640625) disk./.usagePercent‚73.65114647956462 disk.sda2.in‚0 disk.sda.in‚ 7.36328125 disk.rbd3.in‚0 network.recv‚ 3.086483e+06 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625( disk.nvme0c0n1.out‚39825.41064453125 proc.memory.rssMB‚ 2861.78125 cpu‚25.28992797643793& disk.nvme0n1.out‚39825.41064453125 disk.drbd1777.in‚ 1.03515625 disk.sda3.in‚7.3125) proc.memory.percent‚17.46692657470703 disk.nvme0n1.in‚5.3125 disk.nvme0c0n1.in‚5.3125 8]Ù›Â$‚(: çèäÔÊÒÂx gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.218' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.843600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§}߀sÃ$‚(:è çèäÔ¯Žˆ{# disk.dm-0.out‚39825.94677734375 disk.rbd0.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.46692657470703 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0 disk.sda.in‚ 7.36328125 disk.rbd1.out‚ 4.35595703125 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 421.26171875( disk.nvme0c0n1.out‚39825.94677734375 disk.rbd3.out‚0 cpu‚25.28949233988168 disk.rbd2.out‚254.5888671875 disk.rbd4.out‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.sda3.in‚7.3125$ memory_percent‚97.93219566345215 proc.memory.rssMB‚ 2861.78125 network.sent‚ 2.2576407e+07) disk./.usagePercent‚73.65109886216324& disk.nvme0n1.out‚39825.94677734375 disk.rbd0.out‚0 network.recv‚ 3.093695e+06% disk./.usageGB‚1097.4545974731445 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 421.30078125 disk.sda3.out‚ 421.26171875' proc.memory.availableMB‚ 338.7890625 disk.rbd1.in‚0ô’ûAÄ$‚(:… öèäÔ»¹ºx gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.332' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùJ”©mÅ$‚(:â öèäÔ™¨äz disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625' proc.memory.availableMB‚ 338.5546875 proc.memory.rssMB‚ 2861.78125 disk.rbd4.out‚0% disk./.usageGB‚1097.4545974731445 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 disk.rbd2.in‚0 disk.sda3.in‚7.3125% disk.nvme0n1.out‚39826.9013671875) disk./.usagePercent‚73.65109886216324 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 421.8515625) proc.memory.percent‚17.46692657470703 cpu‚25.289087220124628 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 4.35595703125 disk.dm-1.out‚ 421.8515625$ memory_percent‚97.93362617492676 disk.nvme0c0n1.in‚5.3125 network.sent‚ 2.2582348e+07 network.recv‚ 3.097571e+06 disk.rbd2.out‚254.5888671875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0" disk.dm-0.out‚39826.9013671875 disk.dm-1.in‚7.3125 proc.cpu.threads‚17' disk.nvme0c0n1.out‚39826.9013671875 disk.sda.out‚ 421.890625A<}€2Æ$- _wandbruntime‚8819  _runtime‚8819±é¯ ÉÇ$à ' lr-AdamW/pg1‚0.00014051491240175618 ' lr-AdamW/pg2‚0.00014051491240175618 trainer/global_step‚9099 % _timestamp‚1.7884253489255376e+09 _runtime‚8819.675475113 _step‚7452ÕGÎÈ$È ' lr-AdamW/pg1‚0.00014051491240175618 ' lr-AdamW/pg2‚0.00014051491240175618 trainer/global_step‚9099 % _timestamp‚1.7884253489255376e+09 _runtime‚8819.675475113 _step‚745é¤ð;2É$- _wandbruntime‚8819  _runtime‚8819lÓEí'Ê$çO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚360420444322816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360420426319872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚403941110707200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚377318550248960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚386748764071936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358130694734058 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358130454045550 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16897877722624 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 236800966 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16849057146090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16848833496942 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159463592 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167841831 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61268698 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236775296 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167841831 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚360420444322816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17192079910912 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚8819.676389235 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98194894 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61269345 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724352 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16898105926144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚386748740131840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 % _timestamp‚1.7884253489262917e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 404642797 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 236800966 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚403940820042752 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17192346635264 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚341281620548608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 404617125 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341281637587968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 236775296 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚746 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167841829 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377318304042496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚9099 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚377318304042496 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚360420426319872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16898105926144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377318550248960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464ì+Õ_#Ë$ìO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚386748740131840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 % _timestamp‚1.7884253489262917e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 404642797 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 236800966 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚403940820042752 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17192346635264 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚341281620548608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 404617125 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341281637587968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 236775296 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚746 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167841829 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377318304042496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚9099 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚377318304042496 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚360420426319872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16898105926144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377318550248960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚360420444322816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360420426319872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚403941110707200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚377318550248960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚386748764071936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358130694734058 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358130454045550 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16897877722624 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 236800966 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16849057146090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16848833496942 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159463592 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167841831 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61268698 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236775296 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167841831 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚360420444322816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17192079910912 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚8819.676389235 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98194894 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61269345 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724352 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16898105926144êŸ(ÜûÌ$‚(:„ …éäÔ…÷²x gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.928' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7856 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ udÍ$‚(:Ù …éäÔÊÊúz disk.drbd1777.in‚ 1.03515625 disk.rbd3.in‚0 disk.sda3.in‚7.3125% disk.nvme0n1.out‚39827.1474609375 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.rbd2.in‚0 proc.memory.rssMB‚ 2861.78125 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd0.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1097.4629554748535 disk.sda1.out‚0 disk.rbd1.out‚ 4.35595703125 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 network.recv‚ 3.102531e+06" disk.dm-0.out‚39827.1474609375 disk.dm-1.out‚422.25$ memory_percent‚97.93088436126709 disk.sda.in‚ 7.36328125 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.dm-1.in‚7.3125) proc.memory.percent‚17.46692657470703 cpu‚25.288680803874236 network.sent‚ 2.258859e+07 disk.sda.out‚ 422.2890625 disk.sda3.out‚422.25 disk.rbd2.out‚256.3037109375( proc.memory.availableMB‚ 339.00390625) disk./.usagePercent‚73.65165977467078' disk.nvme0c0n1.out‚39827.1474609375 disk.rbd1.in‚0/ÕÉð2Î$- _wandbruntime‚8820  _runtime‚8820 e9lö&Ï$ðM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16899715417088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚403983868730368 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 404686093 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 404660395 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360458764263936 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167859550 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 260419584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚358168829591722 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 236800847 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341317923006464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61275576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚377358725900800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚386789933768192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16899943633920 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159480193 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 _step‚747 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167859550 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚377358479681024 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236826543 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167859548 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16899943633920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17193958902272 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 trainer/global_step‚9099 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98205249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚403984153090048 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16850889545898 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16899715417088 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚358168588903006 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _runtime‚8820.549803951 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚377358725900800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚386789909828096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341317940045824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17194219321856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 $ _timestamp‚1.788425349799258e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61274944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16850665896542 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 404660395 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98205251 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 284359680 ?Yû&Ð$õM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 404686093 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 404660395 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360458764263936 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167859550 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 260419584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚358168829591722 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 236800847 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341317923006464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61275576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚377358725900800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚386789933768192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16899943633920 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159480193 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 _step‚747 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167859550 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚377358479681024 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236826543 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167859548 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16899943633920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17193958902272 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 trainer/global_step‚9099 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98205249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚403984153090048 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16850889545898 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16899715417088 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚358168588903006 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _runtime‚8820.549803951 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚377358725900800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚386789909828096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341317940045824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17194219321856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 $ _timestamp‚1.788425349799258e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61274944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16850665896542 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 404660395 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98205251 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 284359680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16899715417088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚403983868730368 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 404686093 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚379327100382720 % _timestamp‚1.7884253946865149e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160303267 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16990005063168 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168727779 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16940690717298 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168727779 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61590623 _runtime‚8865.436668565 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160303942 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚406092994653696 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚379327346589184 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚379327100382720 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚362337323523072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388807224944640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 238056198 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚406092695600640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16990005063168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388807248884736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 406809649 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 238056198 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚675 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚362337341526016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360037451198066 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360037210509558 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 238081868 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61591296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160303942 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚406092994653696 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚379327346589184 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚379327100382720 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚362337323523072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388807224944640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 238056198 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚406092695600640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16990005063168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388807248884736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 406809649 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 238056198 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚675 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚362337341526016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360037451198066 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360037210509558 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 238081868 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61591296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚379327100382720 % _timestamp‚1.7884253946865149e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160303267 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16990005063168 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168727779 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16940690717298î=Ç52à$- _wandbruntime‚8866  _runtime‚88665dâŠ÷&á$ñM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 406852945 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160320779 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16991614554112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16991842770944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 trainer/global_step‚9149 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚362375679470080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 406827247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98723001 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚379367522241024 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61597778 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 406852945 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚343133045899264 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16942299467750 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 _step‚751 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61597136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚406136024766976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚406135738310144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚379367522241024 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16942523117106 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360075345367014 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 238107445 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 238081749 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 406827247 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚362375679470080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98722999 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 286456832 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160320135 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884253955345361e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168745500 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168745498 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚343133062938624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17287606185984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚379367276021248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168745498 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168745500 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238081749 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 406852945 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160320779 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16991614554112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16991842770944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 trainer/global_step‚9149 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚362375679470080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 406827247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 KEDeviceStatsMonitùpwû†or.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98723001 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚379367522241024 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61597778 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 406852945 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚343133045899264 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16942299467750 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 _step‚751 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61597136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚406136024766976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚406135738310144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚379367522241024 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16942523117106 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360075345367014 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 238107445 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 238081749 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 406827247 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚362375679470080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98722999 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 286456832 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160320135 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884253955345361e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168745500 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168745498 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚343133062938624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17287606185984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚379367276021248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168745498ïIf^2ã$- _wandbruntime‚8866  _runtime‚8866·;û¡]ä$× trainer/global_step‚9149 % _timestamp‚1.7884253955364883e+09 2loss/SRqqt4/assignment_loss‚1.4734601441259478 0loss/FBt1/assignment_loss‚0.6868664265978927 1loss/FBt4/assignment_loss‚0.07052552109558342 0loss/SRqqt1/detection_loss‚2.315222019329667 2loss/SRqqt1/assignment_loss‚1.6503937842790037 /loss/FBt3/assignment_loss‚0.658762963663321 .loss/FRt2/detection_loss‚1.803694098408139 /loss/FBt2/detection_loss‚0.7859884692755129 1loss/FBt2/assignment_loss‚0.17507570494100197 /loss/FRt3/detection_loss‚2.4734808016907084 1loss/SRqqt2/detection_loss‚1.0550086510113694 1loss/SRqqt3/detection_loss‚1.8014939027089698 epoch‚7 _runtime‚8866.286335332 /loss/FRt1/assignment_loss‚3.268261219967495 0loss/FRt3/assignment_loss‚2.5669501288370653 2loss/SRqqt3/assignment_loss‚1.1653500658770402 /loss/FBt1/detection_loss‚2.9987020194530487 /loss/FBt4/detection_loss‚2.3865099814203052 0loss/FRt4/assignment_loss‚2.2486158490180967 2loss/SRqqt2/assignment_loss‚0.8097137730625091 /loss/FRt4/detection_loss‚2.2763076936494575 1loss/SRqqt4/detection_loss‚1.8324012756347656 /loss/FRt1/detection_loss‚2.4836881973526697 &loss/total_loss‚40.858572562951835 _step‚752 0loss/FRt2/assignment_loss‚1.4694073449733644 .loss/FBt3/detection_loss‚2.402692526578903ù‡ïbå$Ü trainer/global_step‚9149 % _timestamp‚1.7884253955364883e+09 2loss/SRqqt4/assignment_loss‚1.4734601441259478 0loss/FBt1/assignment_loss‚0.6868664265978927 1loss/FBt4/assignment_loss‚0.07052552109558342 0loss/SRqqt1/detection_loss‚2.315222019329667 2loss/SRqqt1/assignment_loss‚1.6503937842790037 /loss/FBt3/assignment_loss‚0.658762963663321 .loss/FRt2/detection_loss‚1.803694098408139 /loss/FBt2/detection_loss‚0.7859884692755129 1loss/FBt2/assignment_loss‚0.17507570494100197 /loss/FRt3/detection_loss‚2.4734808016907084 1loss/SRqqt2/detection_loss‚1.0550086510113694 1loss/SRqqt3/detection_loss‚1.8014939027089698 epoch‚7 _runtime‚8866.286335332 /loss/FRt1/assignment_loss‚3.268261219967495 0loss/FRt3/assignment_loss‚2.5669501288370653 2loss/SRqqt3/assignment_loss‚1.1653500658770402 /loss/FBt1/detection_loss‚2.9987020194530487 /loss/FBt4/detection_loss‚2.3865099814203052 0loss/FRt4/assignment_loss‚2.2486158490180967 2loss/SRqqt2/assignment_loss‚0.8097137730625091 /loss/FRt4/detection_loss‚2.2763076936494575 1loss/SRqqt4/detection_loss‚1.8324012756347656 /loss/FRt1/detection_loss‚2.4836881973526697 &loss/total_loss‚40.858572562951835 _step‚752 0loss/FRt2/assignment_loss‚1.4694073449733644 .loss/FBt3/detection_loss‚2.402692526578903ð<¾§”šæ$‚(: ÁéäÔ湯x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚93.105' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.621000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Iˆ„ápç$‚(:å ÁéäÔ´ÔÊ{ disk.sda.out‚ 424.3359375 disk.rbd1.out‚ 4.35595703125 disk.nvme0n1.in‚5.3125 cpu‚25.287069253504708 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.sda2.out‚ 0.0390625# memory_percent‚97.9327917098999% disk./.usageGB‚1097.7133102416992 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 disk.sda2.in‚0 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 network.sent‚ 2.2795401e+07! disk.rbd2.out‚256.55029296875 disk.rbd0.in‚0( proc.memory.availableMB‚ 338.69140625) disk./.usagePercent‚73.66846129313542 disk.nvme0c0n1.in‚5.3125 disk.sda3.out‚ 424.296875 disk.sda3.in‚7.3125& disk.nvme0n1.out‚39936.27001953125# disk.dm-0.out‚39936.27001953125 disk.rbd3.in‚0 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 424.296875 network.recv‚ 3.124598e+06( disk.nvme0c0n1.out‚39936.27001953125 èÜ’›è$‚(: ÐéäÔ¦þ„x gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.727' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.945400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßpŠKpé$‚(:å ÐéäÔ¢ùœ| disk.rbd0.out‚0 disk.sda.in‚ 7.36328125 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 424.984375 disk.nvme0c0n1.in‚5.3125 disk.dm-1.in‚7.3125 disk.drbd1777.out‚0 disk.rbd4.in‚0# disk.dm-0.out‚40036.57080078125 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 cpu‚25.286666843360614& disk.nvme0n1.out‚40036.57080078125 disk.sda2.out‚ 0.0390625% disk./.usageGB‚1097.7117385864258 disk.rbd3.out‚0 disk.sda.out‚ 425.0234375 disk.sda3.in‚7.3125$ memory_percent‚97.93274402618408( disk.nvme0c0n1.out‚40036.57080078125 disk.sda2.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625 disk.nvme0n1.in‚5.3125! disk.rbd2.out‚256.55029296875 network.sent‚ 2.2903524e+07 disk.sda3.out‚ 424.984375) disk./.usagePercent‚73.66835581803132 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.35595703125( proc.memory.availableMB‚ 338.69921875 network.recv‚ 3.13068e+06 disk.rbd0.in‚0iƈ{ê$‚(:„ ßéäÔÏËÅx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.251' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥oyJoë$‚(:ä ßéäÔ¥¡¦{& disk.nvme0n1.out‚40036.82080078125 disk.sda2.out‚ 0.0390625! disk.rbd2.out‚258.26513671875' proc.memory.availableMB‚ 338.6953125 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚40036.82080078125 disk.dm-1.in‚7.3125 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 cpu‚25.286274990482724 disk.rbd1.out‚ 4.35595703125 disk.sda2.in‚0 network.recv‚ 3.135287e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 425.2734375 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.drbd1777.in‚ 1.03515625 disk.sda3.in‚7.3125 disk.dm-0.in‚5.3125 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚40036.82080078125 network.sent‚ 2.2909553e+07% disk./.usageGB‚1097.9627380371094 disk.sda.out‚425.3125 disk.sda3.out‚ 425.2734375 disk.rbd1.in‚0$ memory_percent‚97.93276786804199) disk./.usagePercent‚73.68520060176927 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.rbd4.in‚0ôç¿‹2ì$- _wandbruntime‚8911  _runtime‚8911R|³:Éí$à trainer/global_step‚9199 % _timestamp‚1.7884254402520192e+09 _runtime‚8911.001931646 _step‚753 ' lr-AdamW/pg1‚0.00013182022140911684 ' lr-AdamW/pg2‚0.00013182022140911684µ ìéÎî$È ' lr-AdamW/pg2‚0.00013182022140911684 trainer/global_step‚9199 % _timestamp‚1.7884254402520192e+09 _runtime‚8911.001931646 _step‚753 ' lr-AdamW/pg1‚0.00013182022140911684ñ€&Á2ï$- _wandbruntime‚8911  _runtime‚8911rMì'ð$æO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 239337100 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17081904200192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 169613729 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61916863 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚361944207662074 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚381336142929408 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169613731 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169613729 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚364254238729216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚408245040390656 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 _step‚754 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390865733697536 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 408950829 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 239362770 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚364254220726272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17379039968768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚408244749726208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 $ _timestamp‚1.788425440252811e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17081675996672 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161146609 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99230394 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61916215 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17081675996672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚361943966973566 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 239337100 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 _runtime‚8911.003072104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚381335896722944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17379306693120 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390865709757440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚344911866334208 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚381336142929408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169613731 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 trainer/global_step‚9199 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17032100639358 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 239362770 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚364254238729216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚381335896722944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17379306693120 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390865709757440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚344911866334208 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚381336142929408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169613731 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 trainer/global_step‚9199 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17032100639358 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 239362770 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚364254238729216 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390865733697536 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 408950829 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 239362770 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚364254220726272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17379039968768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚408244749726208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 $ _timestamp‚1.788425440252811e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17081675996672 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161146609 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99230394 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61916215 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17081675996672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚361943966973566 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 239337100 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ò¸Õ'2ò$- _wandbruntime‚8911  _runtime‚8911üC+ö&ó$ðM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239362651 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 239388347 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚408288083904000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309525504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 239362651 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17083513691136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 169631448 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚8911.859344314 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚408287774378496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚364292576673280 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161163412 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚364292558670336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61922663 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17083513691136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169631450 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17380894924800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚361982101831022 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169631448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 409019797 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚381376318581248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚381376072361472 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17381180510208 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17033933038958 $ _timestamp‚1.788425441108971e+09 _step‚755 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0‰hB‚û&ô$õM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 408994099 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 408994099 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99240749 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61923322 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚8911.859344314 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚408287774378496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚364292576673280 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161163412 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚364292558670336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61922663 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17083513691136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169631450 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17380894924800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚361982101831022 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169631448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 409019797 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚381376318581248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚381376072361472 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17381180510208 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17033933038958 $ _timestamp‚1.788425441108971e+09 _step‚755 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239362651 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 239388347 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚408288083904000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309525504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 239362651 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17083513691136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 169631448 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚383344693063168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17472725981184 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor}0_Q .on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366171117929472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚9249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 $ _timestamp‚1.788425485369997e+09 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17123734210566 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 240618002 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚363850964126082 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚383344939269632 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170499681 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 240618002 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346727006266368 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17173803337216 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161989295 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚410396639147008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170499679 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170499681 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 411143353 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99748146 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚366171117929472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240643672 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8956.120132979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚366171135932416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚363850723437574 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99748144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17173575133696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 411117681 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170499679 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17173803337216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚392924218510336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392924194570240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚758 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346726989227008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17173575133696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17123957859714 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚383344693063168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚410396944491520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17472444576768õnÐñ'ƒ%ëO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392924194570240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚758 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346726989227008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17173575133696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17123957859714 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚383344693063168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚410396944491520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17472444576768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚383344939269632 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161988612 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 240643672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62240468 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚383344693063168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17472725981184 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366171117929472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚9249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 $ _timestamp‚1.788425485369997e+09 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17123734210566 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 240618002 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚363850964126082 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚383344939269632 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170499681 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 240618002 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346727006266368 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17173803337216 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161989295 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚410396639147008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170499679 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170499681 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 411143353 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99748146 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚366171117929472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240643672 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8956.120132979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚366171135932416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚363850723437574 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99748144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17173575133696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 411117681 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170499679 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17173803337216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚392924218510336ö5Q>û2„%- _wandbruntime‚8956  _runtime‚8956¨7§Ã÷&…%ñM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 411160951 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚383385114921472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚410439972482048 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17474311272960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17125566610166 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 411186649 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17474584275456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚366209455873536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 trainer/global_step‚9249 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162005950 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚410439675539456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346763291684864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884254862309754e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170517398 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17175412828160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170517398 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 240643553 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚366209455873536 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240669249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚366209473876480 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 411160951 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _runtime‚8956.981734651 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚675 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363888858295030 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚383384868701696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366209473876480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346763308724224 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚383385114921472 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162006627 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392965388206592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17175412828160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170517400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240643553 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 _step‚759 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚675 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363888858295030 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚383384868701696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366209473876480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346763308724224 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚383385114921472 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162006627 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392965388206592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17175412828160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170517400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240643553 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.cBSŠurrent‚299 _step‚759 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 411186649 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17474584275456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚366209455873536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 trainer/global_step‚9249 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162005950 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚410439675539456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346763291684864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884254862309754e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170517398 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17175412828160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170517398 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 240643553 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚366209455873536 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240669249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚366209473876480 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 411160951 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _runtime‚8956.981734651 P2%- _wandbruntime‚9001  _runtime‚9001Ü%ÿÆ‘%À _step‚761 & lr-AdamW/pg1‚0.0001233500381898469 & lr-AdamW/pg2‚0.0001233500381898469 trainer/global_step‚9299 $ _timestamp‚1.788425530529166e+09 _runtime‚9001.279102529GÇ'Ë’%Å & lr-AdamW/pg1‚0.0001233500381898469 & lr-AdamW/pg2‚0.0001233500381898469 trainer/global_step‚9299 $ _timestamp‚1.788425530529166e+09 _runtime‚9001.279102529 _step‚761ùøêt2“%- _wandbruntime‚9001  _runtime‚9001‚ëªï'”%éO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62562855 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100265896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 413310205 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17265702474240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚385353489403392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚368088033135616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17265474270720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412548508547072 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162828749 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚385353735609856 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _step‚762 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚368088033135616 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162829438 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚368088015132672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884255305299237e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62563542 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17215591430922 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171385629 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100265894 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412548809697280 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17565829164032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚365757479901582 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 241924574 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚394982703323136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚394982679383040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚385353489403392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚368088033135616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17265474270720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412548508547072 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162828749 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚385353735609856 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _step‚762 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ]IDeviceStatsMonitor.on_train_batchd•Ë*²_start/active_bytes.large_pool.allocated‚368088033135616 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162829438 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚368088015132672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884255305299237e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62563542 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17215591430922úñ}á2–%- _wandbruntime‚9002  _runtime‚9002°-^ù&—%óM _runtime‚9002.128123402 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 413327803 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 241950151 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 413353501 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚348578431617024 _step‚763 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62569575 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62568940 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17267311965184 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100276251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162845189 % _timestamp‚1.7884255313775158e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚368126353076736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 241924455 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17567694280192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17267540182016 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412591543359488 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17267540182016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171403348 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171403348 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 171403350 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚348578414577664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17267311965184 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100276249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚385393911261696 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 241950151 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚395023873019392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17567967282688 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 413327803 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162845826 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0‚J#Ñþ&˜%øM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 241924455 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚348578431617024 _step‚763 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62569575 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62568940 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17267311965184 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100276251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162845189 % _timestamp‚1.7884255313775158e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚368126353076736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 241924455 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17567694280192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17267540182016 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412591543359488 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17267540182016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171403348 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171403348 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 171403350 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚348578414577664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17267311965184 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100276249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚385393911261696 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 241950151 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚395023873019392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17567967282688 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 413327803 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162845826 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚9002.128123402 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 413327803 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 241950151 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 413353501 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PxÁ %‚(:… æêäÔÙŽ­x gpu.0.gpu‚20.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k¡¹Çj¡%‚(:ß æêäÔ˜Òz disk.drbd1777.out‚0 disk.sda.in‚ 7.36328125 disk.nvme0n1.in‚5.3125% disk.nvme0n1.out‚40733.0341796875 disk.dm-1.out‚ 430.0390625! disk.rbd2.out‚260.45361328125 disk.rbd1.out‚ 4.66259765625 disk.dm-0.in‚5.3125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125 proc.cpu.threads‚17( disk./.usagePercent‚73.7272503512833 disk.sda3.in‚7.3125 disk.rbd1.in‚0" disk.dm-0.out‚40733.0341796875 disk.dm-1.in‚7.3125' disk.nvme0c0n1.out‚40733.0341796875 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 network.recv‚ 3.181181e+06% disk./.usageGB‚1098.5893096923828 disk.rbd0.in‚0 disk.rbd4.in‚0$ memory_percent‚97.93357849121094) proc.memory.percent‚17.46692657470703 cpu‚25.282652372434423 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 430.078125 network.sent‚ 2.3269836e+07 disk.sda1.out‚0$ proc.memory.availableMB‚338.5625 disk.sda3.out‚ 430.0390625 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 proc.memory.rssMB‚ 2861.78125GG¶€2¢%- _wandbruntime‚9046  _runtime‚9046!3É£%à ' lr-AdamW/pg2‚0.00011920135081923968 trainer/global_step‚9349 % _timestamp‚1.7884255757496004e+09 _runtime‚9046.499532833 _step‚765 ' lr-AdamW/pg1‚0.00011920135081923968“ù×eΤ%È ' lr-AdamW/pg1‚0.00011920135081923968 ' lr-AdamW/pg2‚0.00011920135081923968 trainer/global_step‚9349 % _timestamp‚1.7884255757496004e+09 _runtime‚9046.499532833 _step‚765ý¼†Üš2¥%- _wandbruntime‚9046  _runtime‚9046ð r¾ ¦%èO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17357373407744 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17307225002130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚350357252051968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚414700423379968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397041164195840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚766 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172271579 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚370004930338816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚370004912335872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚414700722433024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚367664477054098 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17307001352982 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚387362285743616 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172271579 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 243179806 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 163667639 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_t}0Òò0rain_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17357373407744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 415477057 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚350357235012608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚387362285743616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚370004930338816 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172271581 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397041188135936 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ŸGpoó'§%íO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚350357235012608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚387362285743616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚370004930338816 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172271581 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397041188135936 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17357373407744 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17307225002130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚350357252051968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚414700423379968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397041164195840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚766 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172271579 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚370004930338816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚370004912335872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚414700722433024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚367664477054098 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17307001352982 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚387362285743616 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172271579 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 243179806 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 163667639 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17357373407744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 415477057 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561þîyðö2¨%- _wandbruntime‚9047  _runtime‚90479Gø&©%òM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 415520353 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17308833752582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62889569 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚367702371223046 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 243231053 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚414743773288448 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 _step‚767 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 415494655 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚370043268282880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚370043268282880 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 163684211 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397082333892096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚350393537470464 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17359211102208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9349 _runtime‚9047.354620863 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 243231053 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚387402707601920 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚387402461382144 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172289298 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17359211102208 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172289300 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163683568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17309057401938 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17359439319040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚414743478443008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 $ _timestamp‚1.788425576604009e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚387402707601920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100793999 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17661415456256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17661144550912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172289300 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 243205357 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17359439319040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 415520353 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚367702611911762 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚397082357832192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172289298 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100794001 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚350393554509824 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440¦q{„ª%÷M L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚370043268282880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚370043268282880 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 163684211 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397082333892096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚350393537470464 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17359211102208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9349 _runtime‚9047.354620863 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 243231053 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚387402707601920 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚387402461382144 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172289298 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17359211102208 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172289300 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163683568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17309057401938 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17359439319040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚414743478443008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 $ _timestamp‚1.788425576604009e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚387402707601920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100793999 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17661415456256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17661144550912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172289300 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 243205357 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17359439319040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 415520353 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚367702611911762 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚397082357832192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172289298 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100794001 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚350393554509824 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 415520353 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17308833752582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62889569 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚367702371223046 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 243231053 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚414743773288448 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 _step‚767 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 415494655 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824ÿši¼2«%- _wandbruntime‚9047  _runtime‚9047ûD¦X¬%Ò 0loss/FRt2/assignment_loss‚1.7002037525177003 /loss/FRt2/detection_loss‚1.5513167898403482 0loss/FRt1/assignment_loss‚3.1812670701428463 .loss/FRt3/detection_loss‚2.498210664405379 1loss/SRqqt1/detection_loss‚2.2156932886157716 1loss/SRqqt2/detection_loss‚0.8237608786385793 /loss/FBt4/detection_loss‚2.3388771961132684 _step‚768 #loss/total_loss‚42.601864503879 epoch‚7 % _timestamp‚1.7884255766058662e+09 0loss/FRt4/assignment_loss‚3.4489385034979843 2loss/SRqqt2/assignment_loss‚0.9647834992179504 2loss/SRqqt3/assignment_loss‚1.7330387913367964 2loss/SRqqt4/assignment_loss‚1.7019584853540768 .loss/FBt3/detection_loss‚3.315020263195038 1loss/FBt2/assignment_loss‚0.04882211678414049 /loss/FBt1/detection_loss‚1.9592359388868015 .loss/FBt2/detection_loss‚0.678644531698993 1loss/FBt4/assignment_loss‚0.16416670961189084 1loss/FBt1/assignment_loss‚0.28862263497118573 1loss/SRqqt4/detection_loss‚2.0565806822343307 _runtime‚9047.355700527 1loss/SRqqt1/assignment_loss‚1.763101939111948 .loss/FRt1/detection_loss‚2.486247882246971 /loss/FRt4/detection_loss‚2.3262062795416885 trainer/global_step‚9349 /loss/FRt3/assignment_loss‚3.155796250631643 /loss/FBt3/assignment_loss‚0.522236343473196 0loss/SRqqt3/detection_loss‚1.6791340118104765§D]­%× _step‚768 #loss/total_loss‚42.601864503879 epoch‚7 % _timestamp‚1.7884255766058662e+09 0loss/FRt4/assignment_loss‚3.4489385034979843 2loss/SRqqt2/assignment_loss‚0.9647834992179504 2loss/SRqqt3/assignment_loss‚1.7330387913367964 2loss/SRqqt4/assignment_loss‚1.7019584853540768 .loss/FBt3/detection_loss‚3.315020263195038 1loss/FBt2/assignment_loss‚0.04882211678414049 /loss/FBt1/detection_loss‚1.9592359388868015 .loss/FBt2/detection_loss‚0.678644531698993 1loss/FBt4/assignment_loss‚0.16416670961189084 1loss/FBt1/assignment_loss‚0.28862263497118573 1loss/SRqqt4/detection_loss‚2.0565806822343307 _runtime‚9047.355700527 1loss/SRqqt1/assignment_loss‚1.763101939111948 .loss/FRt1/detection_loss‚2.486247882246971 /loss/FRt4/detection_loss‚2.3262062795416885 trainer/global_step‚9349 /loss/FRt3/assignment_loss‚3.155796250631643 /loss/FBt3/assignment_loss‚0.522236343473196 0loss/SRqqt3/detection_loss‚1.679134011810476 0loss/FRt2/assignment_loss‚1.7002037525177003 /loss/FRt2/detection_loss‚1.5513167898403482 0loss/FRt1/assignment_loss‚3.1812670701428463 .loss/FRt3/detection_loss‚2.498210664405379 1loss/SRqqt1/detection_loss‚2.2156932886157716 1loss/SRqqt2/detection_loss‚0.8237608786385793 /loss/FBt4/detection_loss‚2.3388771961132684€ŽýÒü®%‚(:… õêäÔÿª£y gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þë”"p¯%‚(:å õêäÔ³äñz network.recv‚ 3.186033e+06& disk.nvme0n1.out‚40938.52490234375 disk.dm-0.in‚5.3125 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.sda3.in‚7.3125 disk.rbd1.out‚ 4.66259765625# disk.dm-0.out‚40938.52490234375 disk.sda2.in‚0 disk.rbd3.out‚0) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 disk.dm-1.in‚7.3125! disk.rbd2.out‚260.46923828125( disk./.usagePercent‚73.7391534215878 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 430.453125 network.sent‚ 2.3276129e+07 disk.dm-1.out‚ 430.4140625 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 430.4140625 cpu‚25.282266889064598$ disk./.usageGB‚1098.766674041748 disk.rbd1.in‚0 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚40938.52490234375 disk.drbd1777.in‚ 1.03515625$ memory_percent‚97.93694019317627( proc.memory.availableMB‚ 338.01171875 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd3.in‚0 proc.memory.rssMB‚ 2861.78125‰´õî°%‚(:… „ëäÔÍ÷’x gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.584' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹ !m±%‚(:â „ëäÔåšÀ{ disk.rbd3.out‚0# memory_percent‚97.9360580444336) disk./.usagePercent‚73.73915290957274 disk.rbd0.in‚0 disk.sda.out‚ 431.1796875 disk.rbd1.in‚0 proc.memory.rssMB‚ 2861.78125 network.sent‚ 2.3384199e+07 network.recv‚ 3.192279e+06 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 disk.nvme0n1.in‚5.3125) proc.memory.percent‚17.46692657470703 disk.sda2.in‚0! disk.rbd2.out‚262.66845703125 disk.rbd3.in‚0 disk.sda3.out‚ 431.140625 disk.rbd1.out‚ 4.66259765625# disk.dm-0.out‚41038.19091796875 disk.dm-1.in‚7.3125 disk.rbd4.out‚0 disk.sda.in‚ 7.36328125& disk.nvme0n1.out‚41038.19091796875 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda3.in‚7.3125 cpu‚25.281880613694163 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 431.140625( disk.nvme0c0n1.out‚41038.19091796875 disk.sda1.in‚ 0.05078125% proc.memory.availableMB‚ 338.15625% disk./.usageGB‚1098.7666664123535 disk.dm-0.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0Û,r¡²%‚(:… “ëäԃרx gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þˆ½s³%‚(:è “ëäÔ¬žÛz( disk./.usagePercent‚73.7391040121337 disk.dm-1.in‚7.3125 disk.rbd3.in‚0 disk.sda.out‚ 432.56640625 proc.memory.rssMB‚ 2861.78125 cpu‚25.28150359513206 network.recv‚ 3.196493e+06# disk.dm-0.out‚41342.09716796875 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda1.in‚ 0.05078125% disk./.usageGB‚1098.7659378051758 disk.rbd1.out‚ 4.66259765625 disk.dm-1.out‚ 432.52734375 disk.rbd0.in‚0 disk.sda3.out‚ 432.52734375( proc.memory.availableMB‚ 338.15234375) proc.memory.percent‚17.46692657470703 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125( disk.nvme0c0n1.out‚41342.09716796875 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 network.sent‚ 2.3390214e+07& disk.nvme0n1.out‚41342.09716796875 disk.drbd1777.out‚0# memory_percent‚97.9360818862915 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625! disk.rbd2.out‚262.74658203125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125ü¶dï2´%- _wandbruntime‚9091  _runtime‚9091þè Èµ% trainer/global_step‚9399 $ _timestamp‚1.788425621137841e+09 _runtime‚9091.887747214 _step‚769 ' lr-AdamW/pg1‚0.00011511142307003705 ' lr-AdamW/pg2‚0.00011511142307003705Ñ\Ô Í¶%Ç _step‚769 ' lr-AdamW/pg1‚0.00011511142307003705 ' lr-AdamW/pg2‚0.00011511142307003705 trainer/global_step‚9399 $ _timestamp‚1.788425621137841e+09 _runtime‚9091.887747214ˆÁý¥2·%- _wandbruntime‚9091  _runtime‚9091¼ê ï'¸%éO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17398858573338 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 173157529 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 244460708 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17449500748288 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚416852666209280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 417618237 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 244460708 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 244486378 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17449272544768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63212335 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17752993260544 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 173157529 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389371328290304 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173157531 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173157531 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚369571233518106 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚371921809539072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101301396 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚369570992829598 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚371921827542016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 244486378 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101301394 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚352172374944768 _step‚770 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚399099672948736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚399099649008640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚371921827542016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚371921809539072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389371082083840 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389371328290304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17398858573338 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 173157529 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 244460708 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17449500748288 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚416852666209280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 417618237 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 244460708 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 244486378 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17449272544768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63212335 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17752993260544 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 173157529 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389371328290304 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173157531 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173157531 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚369571233518106 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚371921809539072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101301396 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚369570992829598 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚371921827542016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 244486378 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101301394 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚352172374944768 _step‚770 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚399099672948736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚399099649008640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚371921827542016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚371921809539072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389371082083840 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389371328290304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 417687205 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚416895704332288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164529559 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 173175248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚416895403195392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17400690973146 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 164530228 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚389411257722368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101311751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 417687205 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 173175250 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17400467323790 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399140842644992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17451338456064 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚371960165486080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚771 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚352208660363264 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 244486259 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚369609368375770 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚371960147483136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17754861687296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 417661507 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 244486259 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 trainer/global_step‚9399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚352208677402624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 173175248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101311749 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17754584490496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 244511955 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17451110239232 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 173175250 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚369609127687054qG—üþ&¼%øM L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164529559 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 173175248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚416895403195392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17400690973146 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 164530228 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚389411257722368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101311751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 417687205 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 173175250 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17400467323790 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399140842644992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17451338456064 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚371960165486080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚771 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚352208660363264 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 244486259 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚369609368375770 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚371960147483136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17754861687296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 417661507 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 244486259 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 trainer/global_step‚9399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚352208677402624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 173175248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101311749 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17754584490496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 244511955 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17451110239232 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 173175250 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚369609127687054 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚371960147483136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399140818704896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 417687205 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚416895704332288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561ƒè{÷2½%- _wandbruntime‚9092  _runtime‚9092ä}sZ¾%Ô 1loss/SRqqt1/assignment_loss‚2.092579395540299 /loss/FRt1/detection_loss‚3.0310677136568462 /loss/FBt1/detection_loss‚2.9966326355934143 /loss/FBt3/detection_loss‚2.7872686811855862 _step‚772 /loss/FRt1/assignment_loss‚3.344584177522098 0loss/FRt3/assignment_loss‚2.7826844033072975 /loss/FRt4/detection_loss‚2.4813495195189748 /loss/SRqqt1/detection_loss‚2.51913470414377 epoch‚7 1loss/FBt2/assignment_loss‚0.15665874776014715 1loss/FBt3/assignment_loss‚0.22871818075821335 /loss/FRt2/detection_loss‚1.7247856044966094 /loss/FBt2/detection_loss‚0.6658559770492668 /loss/FRt2/assignment_loss‚1.488721010819921 1loss/SRqqt3/assignment_loss‚1.547332653862446 1loss/FBt1/assignment_loss‚0.11571094783982971 .loss/FRt3/detection_loss‚2.510650332651886 .loss/FBt4/detection_loss‚2.433638322353363 2loss/SRqqt4/assignment_loss‚1.6434511216159196 1loss/SRqqt4/detection_loss‚2.0054667446207493 %loss/total_loss‚43.70280211433731 _runtime‚9092.741213164 1loss/FBt4/assignment_loss‚0.01993214195172186 1loss/SRqqt2/detection_loss‚0.8212542800853649 trainer/global_step‚9399 0loss/FRt4/assignment_loss‚3.2062317728996277 % _timestamp‚1.7884256219913905e+09 2loss/SRqqt2/assignment_loss‚1.1315092914155684 0loss/SRqqt3/detection_loss‚1.967583753688391¾ÖáW_¿%Ù /loss/FRt1/assignment_loss‚3.344584177522098 0loss/FRt3/assignment_loss‚2.7826844033072975 /loss/FRt4/detection_loss‚2.4813495195189748 /loss/SRqqt1/detection_loss‚2.51913470414377 epoch‚7 1loss/FBt2/assignment_loss‚0.15665874776014715 1loss/FBt3/assignment_loss‚0.22871818075821335 /loss/FRt2/detection_loss‚1.7247856044966094 /loss/FBt2/detection_loss‚0.6658559770492668 /loss/FRt2/assignment_loss‚1.488721010819921 1loss/SRqqt3/assignment_loss‚1.547332653862446 1loss/FBt1/assignment_loss‚0.11571094783982971 .loss/FRt3/detection_loss‚2.510650332651886 .loss/FBt4/detection_loss‚2.433638322353363 2loss/SRqqt4/assignment_loss‚1.6434511216159196 1loss/SRqqt4/detection_loss‚2.0054667446207493 %loss/total_loss‚43.70280211433731 _runtime‚9092.741213164 1loss/FBt4/assignment_loss‚0.01993214195172186 1loss/SRqqt2/detection_loss‚0.8212542800853649 trainer/global_step‚9399 0loss/FRt4/assignment_loss‚3.2062317728996277 % _timestamp‚1.7884256219913905e+09 2loss/SRqqt2/assignment_loss‚1.1315092914155684 0loss/SRqqt3/detection_loss‚1.967583753688391 1loss/SRqqt1/assignment_loss‚2.092579395540299 /loss/FRt1/detection_loss‚3.0310677136568462 /loss/FBt1/detection_loss‚2.9966326355934143 /loss/FBt3/detection_loss‚2.7872686811855862 _step‚772„.&ðnŽÀ%‚(:ƒ ¢ëäÔô¥¦x gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04føoÁ%‚(:ä ¢ëäÔçÌz) disk./.usagePercent‚73.73923303993095 disk.sda3.in‚7.3125% disk.nvme0n1.out‚41442.0380859375 disk.dm-1.out‚ 433.3515625 disk.rbd3.in‚0 disk.rbd3.out‚0 cpu‚25.281123935240718 disk.dm-0.in‚5.3125" disk.dm-0.out‚41442.0380859375 disk.sda2.out‚ 0.0390625 disk.sda1.out‚0 proc.memory.rssMB‚ 2861.78125% disk./.usageGB‚1098.7678604125977 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0' disk.nvme0c0n1.out‚41442.0380859375 disk.sda.out‚ 433.390625 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125$ memory_percent‚97.93741703033447 network.sent‚ 2.3396071e+07! disk.rbd2.out‚262.74658203125) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 disk.nvme0n1.in‚5.3125 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0( proc.memory.availableMB‚ 337.93359375 disk.rbd1.out‚ 4.66259765625 disk.rbd0.out‚0 network.recv‚ 3.200926e+06 disk.sda3.out‚ 433.3515625 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.sda.in‚ 7.36328125ಷ5Â%‚(:„ ±ëäÔÙ÷ªx gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚97.112' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.4224 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•Á°ånÃ%‚(:ã ±ëäÔì·õ{) disk./.usagePercent‚73.73923508799123 disk.rbd1.out‚ 4.66259765625 disk.sda2.out‚ 0.0390625" disk.dm-0.out‚41442.3349609375 disk.nvme0c0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 433.9453125 disk.rbd3.out‚0( proc.memory.availableMB‚ 338.03515625 proc.memory.rssMB‚ 2861.78125 network.sent‚ 2.3505039e+07% disk./.usageGB‚1098.7678909301758 disk.rbd2.out‚262.8525390625 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 433.984375 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd4.out‚0' disk.nvme0c0n1.out‚41442.3349609375 proc.cpu.threads‚17 disk.nvme0n1.in‚5.3125 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125$ memory_percent‚97.93679714202881) proc.memory.percent‚17.46692657470703 cpu‚25.280737339026324 network.recv‚ 3.208758e+06% disk.nvme0n1.out‚41442.3349609375 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 433.9453125 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda3.in‚7.3125Wõ¶BÄ%‚(:… ÀëäÔñ‹µx gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü‡hÅ%‚(:Ý ÀëäÔï™ßz disk.sda.in‚ 7.36328125 disk.rbd0.in‚0 disk.rbd1.out‚ 4.66259765625 disk.nvme0n1.in‚5.3125 disk.dm-0.out‚41541.69140625 network.recv‚ 3.212397e+06# disk.nvme0n1.out‚41541.69140625 disk.rbd2.in‚0 proc.memory.rssMB‚ 2861.78125 disk.rbd2.out‚262.8525390625 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0% disk./.usageGB‚1098.7680320739746 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625 cpu‚25.280338994011935% disk.nvme0c0n1.out‚41541.69140625 disk.sda2.out‚ 0.0390625& disk./.usagePercent‚73.73924456027 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.dm-1.out‚ 434.51171875 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 2.3510349e+07 disk.sda3.in‚7.3125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 434.55078125# memory_percent‚97.9374647140503 proc.cpu.threads‚17 disk.sda3.out‚ 434.51171875 disk.rbd1.in‚0 disk.rbd4.out‚0( proc.memory.availableMB‚ 337.92578125) proc.memory.percent‚17.46692657470703ü! 2Æ%- _wandbruntime‚9137  _runtime‚9137÷ÓÿØÅÇ%¿ % lr-AdamW/pg1‚0.000111081107277052 % lr-AdamW/pg2‚0.000111081107277052 trainer/global_step‚9449 % _timestamp‚1.7884256664092886e+09 _runtime‚9137.159215593 _step‚773:ÛÊÈ%Ä % lr-AdamW/pg1‚0.000111081107277052 % lr-AdamW/pg2‚0.000111081107277052 trainer/global_step‚9449 % _timestamp‚1.7884256664092886e+09 _runtime‚9137.159215593 _step‚773…†¬2É%- _wandbruntime‚9137  _runtime‚9137’éÊ%èO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 245767280 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165354502 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101819144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚391380124630528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391379878424064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17490492144546 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚391380124630528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353987480798208 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 419785089 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17541171681792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419004329765376 J´¿Àllocated‚ 419810761 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 245741610 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚401158157761536 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17846195943936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 245767280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 trainer/global_step‚9449 _runtime‚9137.160172876 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174043479 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚391379878424064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419004624624128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚373838706742272 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚373838724745216 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17541399885312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63535358 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 _step‚774 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174043481 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101819146 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 245741610 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17541171681792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚401158133821440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174043481 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63536036 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚373838724745216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚373838706742272 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 $ _timestamp‚1.788425666410039e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17541399885312oX´:ó'Ë%íO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165354502 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101819144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚391380124630528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391379878424064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17490492144546 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚391380124630528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353987480798208 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 419785089 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17541171681792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419004329765376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 245767280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 trainer/global_step‚9449 _runtime‚9137.160172876 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174043479 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚391379878424064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419004624624128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚373838706742272 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚373838724745216 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17541399885312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63535358 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 _step‚774 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174043481 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101819146 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 245741610 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17541171681792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚401158133821440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174043481 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63536036 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚373838724745216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚373838706742272 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 $ _timestamp‚1.788425666410039e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17541399885312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 245767280 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168†5y Ÿ2Ì%- _wandbruntime‚9138  _runtime‚9138“M2Ëø&Í%òM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚391420054062592 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174061198 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373877062689280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚419047383288320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 245792857 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 419854057 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _step‚775 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373877044686336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17543009376256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401199303517696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚401199327457792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 245767161 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373877044686336 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165371313 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 165371987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174061198 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17543237593088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63541814 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 419854057 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17848079770624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚371515884151062 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚371516124839778 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17543009376256 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9138.026880883 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101829501 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17492100894998 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391420300282368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101829499 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354023800295424 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 245767161 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373877062689280 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17492324544354 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 419828359 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚419047680230912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174061200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168•wX»Î%÷M F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚401199327457792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 245767161 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373877044686336 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165371313 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 165371987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174061198 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17543237593088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63541814 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 419854057 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17848079770624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚371515884151062 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚371516124839778 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17543009376256 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9138.026880883 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101829501 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17492100894998 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391420300282368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101829499 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354023800295424 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 245767161 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373877062689280 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17492324544354 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 419828359 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚419047680230912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174061200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚391420054062592 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174061198 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373877062689280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚419047383288320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 245792857 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 419854057 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _step‚775 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373877044686336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17543009376256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401199303517696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496‡A”Õ2Ï%- _wandbruntime‚9138  _runtime‚9138tk>YÐ%Ó 0loss/SRqqt4/detection_loss‚1.951497547474562 0loss/FRt3/assignment_loss‚2.9790432424772355 0loss/FBt3/assignment_loss‚0.1200745202586404 /loss/FBt2/detection_loss‚0.4097207594154911 2loss/SRqqt2/assignment_loss‚0.6008693194144392 0loss/FBt2/assignment_loss‚0.2095077546149315 /loss/FBt4/assignment_loss‚0.369248232305316 /loss/FRt1/detection_loss‚2.6744801770699653 epoch‚7 2loss/SRqqt1/assignment_loss‚1.2645607819923987 2loss/SRqqt3/assignment_loss‚1.7018117968864896 0loss/SRqqt2/detection_loss‚0.796547597376247 /loss/FBt1/detection_loss‚2.1129687517881393 % _timestamp‚1.7884256672781916e+09 _runtime‚9138.028040548 0loss/FRt1/assignment_loss‚2.8174730225613245 .loss/FBt4/detection_loss‚2.374533074242728 trainer/global_step‚9449 /loss/FRt4/assignment_loss‚2.614788220042274 2loss/SRqqt4/assignment_loss‚1.0698881666815163 1loss/FBt1/assignment_loss‚0.17830144801351935 /loss/FRt2/detection_loss‚1.3076432167435996 0loss/FRt2/assignment_loss‚0.9840710498392582 %loss/total_loss‚39.52226503393421 _step‚776 /loss/FRt3/detection_loss‚2.4190260689883005 .loss/FBt3/detection_loss‚3.356313079595566 .loss/FRt4/detection_loss‚2.359384842635707 1loss/SRqqt1/detection_loss‚2.8642264810892253 1loss/SRqqt3/detection_loss‚1.9862858824273373Ï°Ò%^Ñ%Ø /loss/FRt2/detection_loss‚1.3076432167435996 0loss/FRt2/assignment_loss‚0.9840710498392582 %loss/total_loss‚39.52226503393421 _step‚776 /loss/FRt3/detection_loss‚2.4190260689883005 .loss/FBt3/detection_loss‚3.356313079595566 .loss/FRt4/detection_loss‚2.359384842635707 1loss/SRqqt1/detection_loss‚2.8642264810892253 1loss/SRqqt3/detection_loss‚1.9862858824273373 0loss/SRqqt4/detection_loss‚1.951497547474562 0loss/FRt3/assignment_loss‚2.9790432424772355 0loss/FBt3/assignment_loss‚0.1200745202586404 /loss/FBt2/detection_loss‚0.4097207594154911 2loss/SRqqt2/assignment_loss‚0.6008693194144392 0loss/FBt2/assignment_loss‚0.2095077546149315 /loss/FBt4/assignment_loss‚0.369248232305316 /loss/FRt1/detection_loss‚2.6744801770699653 epoch‚7 2loss/SRqqt1/assignment_loss‚1.2645607819923987 2loss/SRqqt3/assignment_loss‚1.7018117968864896 0loss/SRqqt2/detection_loss‚0.796547597376247 /loss/FBt1/detection_loss‚2.1129687517881393 % _timestamp‚1.7884256672781916e+09 _runtime‚9138.028040548 0loss/FRt1/assignment_loss‚2.8174730225613245 .loss/FBt4/detection_loss‚2.374533074242728 trainer/global_step‚9449 /loss/FRt4/assignment_loss‚2.614788220042274 2loss/SRqqt4/assignment_loss‚1.0698881666815163 1loss/FBt1/assignment_loss‚0.17830144801351935ˆÃÇJ ™Ò%‚(:Ž ÏëäÔËþŒy gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.751' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.750200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04,ÏiÓ%‚(:Þ ÏëäÔý÷¶{ disk.dm-0.in‚5.3125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125% disk./.usageGB‚1098.7679138183594 disk.sda2.in‚0 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 435.07421875 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625$ memory_percent‚97.93665409088135 proc.memory.rssMB‚ 2861.78125 network.sent‚ 2.3516397e+07 disk.sda3.in‚7.3125 disk.dm-0.out‚ 41640.8359375$ disk.nvme0c0n1.out‚ 41640.8359375 cpu‚25.279985753064583 disk.sda3.out‚ 435.07421875( proc.memory.availableMB‚ 338.05859375) proc.memory.percent‚17.46692657470703 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda.out‚ 435.11328125 disk.nvme0n1.in‚5.3125 disk.rbd2.out‚264.5673828125 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 4.66259765625 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0 network.recv‚ 3.217114e+06 disk.rbd1.in‚0" disk.nvme0n1.out‚ 41640.8359375 disk.rbd4.out‚0) disk./.usagePercent‚73.73923662403644 disk.sda1.out‚0 proc.cpu.threads‚17ÁSÔ%‚(:… ÞëäÔ죗x gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èþt‘nÕ%‚(:ã ÞëäÔ¼×ñz disk.dm-1.out‚ 435.41796875 disk.sda1.out‚0 disk.sda.out‚ 435.45703125 proc.memory.rssMB‚ 2861.78125 disk.sda2.in‚0 disk.rbd2.out‚264.5673828125 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd1.out‚ 4.66259765625 proc.cpu.threads‚17 disk.dm-0.in‚5.3125 disk.dm-1.in‚7.3125' disk.nvme0c0n1.out‚41740.0126953125 disk.rbd1.in‚0% disk./.usageGB‚1098.7679824829102 disk.nvme0n1.in‚5.3125" disk.dm-0.out‚41740.0126953125 disk.sda1.in‚ 0.05078125 network.sent‚ 2.3625275e+07 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.sda.in‚ 7.36328125$ proc.memory.availableMB‚338.0625) proc.memory.percent‚17.46692657470703) disk./.usagePercent‚73.73924123217205 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 435.41796875% disk.nvme0n1.out‚41740.0126953125 disk.rbd0.in‚0 disk.drbd1777.out‚0 cpu‚25.279586424477046 network.recv‚ 3.224018e+06 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0$ memory_percent‚97.93663024902344‚^Ö%‚(:… íëäÔ€“‡x gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘)†:q×%‚(:æ íëäԙħ{ disk.rbd4.out‚0% disk./.usageGB‚1098.7679748535156 disk.dm-1.out‚ 436.08984375 disk.sda.out‚ 436.12890625 disk.sda3.out‚ 436.08984375 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚41839.1494140625 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.66259765625 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda3.in‚7.3125 disk.dm-0.in‚5.3125$ memory_percent‚97.93810844421387 proc.cpu.threads‚17 disk.nvme0c0n1.in‚5.3125 disk.sda.in‚ 7.36328125 disk.sda2.in‚0 network.sent‚ 2.3631569e+07 network.recv‚ 3.228712e+06 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 disk.rbd2.in‚0 disk.nvme0n1.in‚5.3125% disk.nvme0n1.out‚41839.1494140625' proc.memory.availableMB‚ 337.8203125 cpu‚25.279209970815824) disk./.usagePercent‚73.73924072015697 disk.dm-1.in‚7.3125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚264.6572265625" disk.dm-0.out‚41839.1494140625¹™]2Ø%- _wandbruntime‚9182  _runtime‚9182 ʦÉÙ%à _runtime‚9182.292016423 _step‚777 ' lr-AdamW/pg1‚0.00010711124335205502 ' lr-AdamW/pg2‚0.00010711124335205502 trainer/global_step‚9499 % _timestamp‚1.7884257115421042e+09C™·ÎÚ%È _runtime‚9182.292016423 _step‚777 ' lr-AdamW/pg1‚0.00010711124335205502 ' lr-AdamW/pg2‚0.00010711124335205502 trainer/global_step‚9499 % _timestamp‚1.7884257115421042e+09‰Í2Û%- _wandbruntime‚9182  _runtime‚9182´vH¬ï'Ü%éO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373384505757614 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17581902066606 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚375755621948416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17939903582720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17582125715754 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421156245006848 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166197359 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17633070818816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63860463 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 174929429 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102336896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403216642574336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 247048182 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403216618634240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102336894 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174929431 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17633299022336 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 247022512 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63859809 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 421951941 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚393388674764288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17939626372608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373384746446122 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174929431 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚375755621948416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17633070818816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 % _timestamp‚1.7884257115428083e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚393388674764288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚375755603945472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17633299022336 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 166196703 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚355802620730368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 trainer/global_step‚9499 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421156546157056×òï‰ô'Ý%îO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 _step‚778 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚375755603945472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 247022512 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63859809 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 421951941 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚393388674764288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17939626372608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373384746446122 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174929431 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚375755621948416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17633070818816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 % _timestamp‚1.7884257115428083e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚393388674764288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚375755603945472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17633299022336 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 166196703 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚355802620730368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 trainer/global_step‚9499 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421156546157056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373384505757614 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17581902066606 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚375755621948416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17939903582720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17582125715754 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421156245006848 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166197359 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17633070818816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63860463 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 174929429 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102336896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403216642574336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 247048182 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403216618634240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102336894 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174929431 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17633299022336 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166213426 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 247073759 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚9499 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 422020909 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403257788330496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMon(¸º‚³"itor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166214060 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚355838906148864 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17583734466206 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚393429096622592 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174947150 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102347251 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 247048063 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403257812270592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 422020909 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174947150 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63866809 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884257124099302e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174947148 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 247048063 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚393428850402816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚393429096622592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚355838923188224 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17583958115562 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17941493045248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 _step‚779 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚375793959892480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚375793959892480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421199578318336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421199281375744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚373422881303786 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 _runtime‚9183.160779801 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17635136730112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 421995211 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚375793941889536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17634908513280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚375793941889536 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17634908513280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 422020909 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403257788330496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166214060 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚355838906148864 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17583734466206 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚393429096622592 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174947150 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102347251 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 247048063 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403257812270592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 422020909 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174947150 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63866809 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884257124099302e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174947148 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 247048063 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚393428850402816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚393429096622592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚355838923188224 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17583958115562 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17941493045248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 _step‚779 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚375793959892480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚375793959892480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421199578318336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421199281375744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚373422881303786 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 _runtime‚9183.160779801 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17635136730112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 421995211 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚375793941889536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17634908513280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚375793941889536 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17634908513280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166213426 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 247073759 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚9499 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608‹¢#÷2á%- _wandbruntime‚9183  _runtime‚9183ë,Š^â%Ø -loss/FRt4/detection_loss‚2.32475856644042 .loss/FBt1/detection_loss‚2.457695823449355 % _timestamp‚1.7884257124121175e+09 2loss/SRqqt4/assignment_loss‚1.0118346439000774 /loss/FRt3/detection_loss‚2.6253164960787845 /loss/FBt2/detection_loss‚0.5201927461419944 &loss/total_loss‚40.366196558620885 2loss/SRqqt2/assignment_loss‚0.9646780561445885 1loss/SRqqt3/detection_loss‚1.6889881657541923 0loss/SRqqt4/detection_loss‚2.133288411066887 2loss/FBt4/assignment_loss‚0.013615212141303346 /loss/FRt1/detection_loss‚2.4647671985764834 0loss/FRt1/assignment_loss‚3.2370807389880336 1loss/FBt1/assignment_loss‚0.10064144398606564 /loss/FRt2/detection_loss‚1.4594878066397672 1loss/SRqqt1/detection_loss‚2.4612630865266247 epoch‚7 trainer/global_step‚9499 0loss/FRt2/assignment_loss‚1.2384432362658637 1loss/FBt3/assignment_loss‚0.13856146104323366 1loss/SRqqt2/detection_loss‚0.8184998948389376 .loss/FBt3/detection_loss‚2.591256300608317 2loss/SRqqt3/assignment_loss‚1.3665499660133753 .loss/FBt4/detection_loss‚3.452330470085144 _runtime‚9183.161950887 _step‚780 /loss/FRt3/assignment_loss‚3.072188436985016 0loss/FRt4/assignment_loss‚2.8049163437904197 2loss/SRqqt1/assignment_loss‚1.3110320909369377 1loss/FBt2/assignment_loss‚0.10880996221906346'Hkcã%Ý trainer/global_step‚9499 0loss/FRt2/assignment_loss‚1.2384432362658637 1loss/FBt3/assignment_loss‚0.13856146104323366 1loss/SRqqt2/detection_loss‚0.8184998948389376 .loss/FBt3/detection_loss‚2.591256300608317 2loss/SRqqt3/assignment_loss‚1.3665499660133753 .loss/FBt4/detection_loss‚3.452330470085144 _runtime‚9183.161950887 _step‚780 /loss/FRt3/assignment_loss‚3.072188436985016 0loss/FRt4/assignment_loss‚2.8049163437904197 2loss/SRqqt1/assignment_loss‚1.3110320909369377 1loss/FBt2/assignment_loss‚0.10880996221906346 -loss/FRt4/detection_loss‚2.32475856644042 .loss/FBt1/detection_loss‚2.457695823449355 % _timestamp‚1.7884257124121175e+09 2loss/SRqqt4/assignment_loss‚1.0118346439000774 /loss/FRt3/detection_loss‚2.6253164960787845 /loss/FBt2/detection_loss‚0.5201927461419944 &loss/total_loss‚40.366196558620885 2loss/SRqqt2/assignment_loss‚0.9646780561445885 1loss/SRqqt3/detection_loss‚1.6889881657541923 0loss/SRqqt4/detection_loss‚2.133288411066887 2loss/FBt4/assignment_loss‚0.013615212141303346 /loss/FRt1/detection_loss‚2.4647671985764834 0loss/FRt1/assignment_loss‚3.2370807389880336 1loss/FBt1/assignment_loss‚0.10064144398606564 /loss/FRt2/detection_loss‚1.4594878066397672 1loss/SRqqt1/detection_loss‚2.4612630865266247 epoch‚7Œi;û}šä%‚(: üëäԜީx gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.751' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.750200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡'¯må%‚(:â üëäÔàüþz disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125 disk.sda3.out‚ 437.078125$ memory_percent‚97.93808460235596 cpu‚25.278838848824837 disk.rbd1.out‚ 4.66259765625 disk.dm-1.out‚ 437.078125 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 network.sent‚ 2.3636879e+07% disk.nvme0n1.out‚41840.0322265625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.sda3.in‚7.3125 network.recv‚ 3.23241e+06 disk.nvme0n1.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚ 437.1171875 disk.rbd1.in‚0" disk.dm-0.out‚41840.0322265625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 337.82421875) disk./.usagePercent‚73.73927707322684% disk./.usageGB‚1098.7685165405273! disk.rbd2.out‚264.67333984375' disk.nvme0c0n1.out‚41840.0322265625 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 proc.cpu.threads‚17 disk.dm-0.in‚5.3125 disk.sda2.out‚ 0.0390625ÕÍZ)æ%‚(:… ‹ìäÔžÔÂx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚078.kç%‚(:à ‹ìäÔŸùz disk.rbd3.in‚0! disk.rbd2.out‚264.67333984375 disk.drbd1777.out‚0) disk./.usagePercent‚73.73927502516656% disk./.usageGB‚1098.7684860229492 disk.dm-1.out‚ 437.46875 disk.sda1.out‚0 disk.sda.in‚ 7.36328125 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0$ memory_percent‚97.93756008148193 disk.sda2.in‚0 disk.sda1.in‚ 0.05078125 disk.dm-0.in‚5.3125 disk.sda3.in‚7.3125 disk.rbd1.out‚ 4.66259765625 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 437.5078125 disk.sda3.out‚ 437.46875% disk.nvme0n1.out‚41939.0947265625" disk.dm-0.out‚41939.0947265625 disk.drbd1777.in‚ 1.03515625 network.sent‚ 2.3745576e+07 network.recv‚ 3.239562e+06 disk.sda2.out‚ 0.0390625 disk.nvme0n1.in‚5.3125 cpu‚25.27846748945114 proc.cpu.threads‚17' disk.nvme0c0n1.out‚41939.0947265625( proc.memory.availableMB‚ 337.91015625qsÜùŽè%‚(:ƒ šìäÔç–x gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.37' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î¾á2né%‚(:ã šìäÔÑþÏ{! disk.rbd2.out‚264.75146484375 disk.nvme0c0n1.in‚5.3125( disk./.usagePercent‚73.7392737451289 disk.rbd3.out‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 438.046875$ memory_percent‚97.93763160705566 proc.memory.rssMB‚ 2861.78125 cpu‚25.278066243564016 network.sent‚ 2.3751508e+07 disk.sda.in‚ 7.36328125 disk.sda1.in‚ 0.05078125# disk.dm-0.out‚42039.00537109375 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0) proc.memory.percent‚17.46692657470703 disk.rbd1.out‚ 4.66259765625 network.recv‚ 3.244063e+06 disk.rbd3.in‚0 disk.sda.out‚ 438.0859375 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125' proc.memory.availableMB‚ 337.8984375 proc.cpu.threads‚17$ disk./.usageGB‚1098.768466949463 disk.rbd1.in‚0 disk.dm-1.in‚7.3125 disk.rbd2.in‚0( disk.nvme0c0n1.out‚42039.00537109375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚42039.00537109375 disk.dm-1.out‚ 438.046875åî±2ê%- _wandbruntime‚9227  _runtime‚9227ÖˆaÉë%à _runtime‚9227.392281118 _step‚781 ' lr-AdamW/pg1‚0.00010320265860873799 ' lr-AdamW/pg2‚0.00010320265860873799 trainer/global_step‚9549 % _timestamp‚1.7884257566423845e+09=Ë^Îì%È ' lr-AdamW/pg1‚0.00010320265860873799 ' lr-AdamW/pg2‚0.00010320265860873799 trainer/global_step‚9549 % _timestamp‚1.7884257566423845e+09 _runtime‚9227.392281118 _step‚781ë$„«2í%- _wandbruntime‚9227  _runtime‚9227#íá€óî%èO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175815381 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚405275127387136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 $ _timestamp‚1.788425756643073e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚423308482895360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377672519151616 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102854644 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 424144465 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 167039077 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377672501148672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚375291262221622 trainer/global_step‚9549 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167039771 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64184433 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 _step‚782 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17725198159360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚395397717310976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.sma´‹2?ûll_pool.current‚692 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚375291502910130 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64185125 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚357617743623168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 424118793 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18033080395264 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚395397471104512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377672501148672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚9227.393171525 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17725198159360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚357617726583808 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102854646 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17673535637814 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚395397471104512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175815381 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 248303414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17724969955840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚423308183842304 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18033355508224 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175815379 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 248329084Iw¿œó'ï%íO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377672501148672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚9227.393171525 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17725198159360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚357617726583808 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102854646 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17673535637814 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚395397471104512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175815381 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 248303414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17724969955840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚423308183842304 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18033355508224 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175815379 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 248329084 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175815381 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚405275127387136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 $ _timestamp‚1.788425756643073e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚423308482895360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377672519151616 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102854644 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 424144465 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 167039077 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377672501148672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚375291262221622 trainer/global_step‚9549 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167039771 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64184433 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 _step‚782 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17725198159360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚395397717310976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚375291502910130 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64185125 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚357617743623168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 424118793 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18033080395264 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚395397471104512Ž¤aW¡2ð%- _wandbruntime‚9228  _runtime‚9228[Í›ù&ñ%óM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚395437892962816 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17727035867136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175833100 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102864999 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17675591686770 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175833098 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167056764 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 424162063 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 424187761 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102865001 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚375329637767794 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚395437646743040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _step‚783 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377710857095680 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17726807650304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚405316297083392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167056080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚357654046081024 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884257575045345e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17727035867136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚405316273143296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377710839092736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚423351517774848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 248328965 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚377710839092736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17675368037414 _runtime‚9228.255284665 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 424187761 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18034947688960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 424162063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64191763 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64191081 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175833098 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175833100 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 248354661 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377710857095680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18035220691456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚357654029041664 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299:§cbÄò%øM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17727035867136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚405316273143296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377710839092736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚423351517774848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 248328965 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚377710839092736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17675368037414 _runtime‚9228.255284665 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 424187761 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18034947688960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 424162063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64191763 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64191081 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175833098 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175833100 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 248354661 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377710857095680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18035220691456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚357654029041664 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚395437892962816 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17727035867136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175833100 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102864999 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17675591686770 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175833098 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167056764 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 424162063 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 424187761 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102865001 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚375329637767794 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚395437646743040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _step‚783 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377710857095680 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17726807650304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚405316297083392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167056080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚357654046081024 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884257575045345e+09ÐÎë2ó%- _wandbruntime‚9228  _runtime‚9228Å4È]ô%× 1loss/SRqqt3/detection_loss‚1.7467773285508157 /loss/FBt3/detection_loss‚2.6446557343006134 .loss/FRt3/detection_loss‚2.208416532145606 1loss/SRqqt4/detection_loss‚1.8342602859843862 % _timestamp‚1.7884257575064812e+09 /loss/FRt1/assignment_loss‚3.038815396588023 /loss/FRt2/assignment_loss‚1.372374022548849 0loss/FRt4/assignment_loss‚2.9136650525033474 2loss/SRqqt3/assignment_loss‚1.2480921855568885 2loss/SRqqt4/assignment_loss‚1.3188494332812049 .loss/FRt4/detection_loss‚2.263513707919882 1loss/SRqqt1/detection_loss‚2.4911668320496876 2loss/SRqqt2/assignment_loss‚0.7099736363268816 1loss/FBt1/assignment_loss‚0.06857522829113805 .loss/FRt1/detection_loss‚2.438965330400118 /loss/FRt2/detection_loss‚1.8924953298727898 1loss/SRqqt2/detection_loss‚1.0899302867742686 .loss/FBt4/detection_loss‚3.893382954597473 %loss/total_loss‚40.97282368200075 epoch‚7 1loss/FBt4/assignment_loss‚0.36545018133474516 0loss/FRt3/assignment_loss‚2.3863106383217705 _runtime‚9228.256299356 _step‚784 2loss/SRqqt1/assignment_loss‚1.4520849615335465 1loss/FBt3/assignment_loss‚0.15311342984205112 /loss/FBt1/detection_loss‚2.8375377357006073 1loss/FBt2/assignment_loss‚0.12291178005074227 /loss/FBt2/detection_loss‚0.4815056775253089 trainer/global_step‚9549MOQbõ%Ü 2loss/SRqqt2/assignment_loss‚0.7099736363268816 1loss/FBt1/assignment_loss‚0.06857522829113805 .loss/FRt1/detection_loss‚2.438965330400118 /loss/FRt2/detection_loss‚1.8924953298727898 1loss/SRqqt2/detection_loss‚1.0899302867742686 .loss/FBt4/detection_loss‚3.893382954597473 %loss/total_loss‚40.97282368200075 epoch‚7 1loss/FBt4/assignment_loss‚0.36545018133474516 0loss/FRt3/assignment_loss‚2.3863106383217705 _runtime‚9228.256299356 _step‚784 2loss/SRqqt1/assignment_loss‚1.4520849615335465 1loss/FBt3/assignment_loss‚0.15311342984205112 /loss/FBt1/detection_loss‚2.8375377357006073 1loss/FBt2/assignment_loss‚0.12291178005074227 /loss/FBt2/detection_loss‚0.4815056775253089 trainer/global_step‚9549 1loss/SRqqt3/detection_loss‚1.7467773285508157 /loss/FBt3/detection_loss‚2.6446557343006134 .loss/FRt3/detection_loss‚2.208416532145606 1loss/SRqqt4/detection_loss‚1.8342602859843862 % _timestamp‚1.7884257575064812e+09 /loss/FRt1/assignment_loss‚3.038815396588023 /loss/FRt2/assignment_loss‚1.372374022548849 0loss/FRt4/assignment_loss‚2.9136650525033474 2loss/SRqqt3/assignment_loss‚1.2480921855568885 2loss/SRqqt4/assignment_loss‚1.3188494332812049 .loss/FRt4/detection_loss‚2.263513707919882 1loss/SRqqt1/detection_loss‚2.4911668320496876Ťö%‚(:„ ©ìäÔÚ„‚x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚98.344' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.6688 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢€ào÷%‚(:ä ©ìäÔ¾ÕÔz disk.sda1.in‚ 0.05078125( proc.memory.availableMB‚ 337.73828125 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-0.in‚5.3125 disk.rbd2.in‚0 disk.rbd4.out‚0 cpu‚25.277729483039217& disk.nvme0n1.out‚42138.13525390625! disk.rbd2.out‚266.26318359375 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0$ memory_percent‚97.93860912322998 disk.drbd1777.in‚ 1.03515625 network.sent‚ 2.3757796e+07 network.recv‚ 3.249191e+06 disk.sda3.in‚7.3125# disk.dm-0.out‚42138.13525390625 proc.memory.rssMB‚ 2861.78125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚42138.13525390625 disk.nvme0n1.in‚5.3125 disk.rbd3.in‚0 disk.sda.out‚ 438.7109375 disk.sda2.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.in‚ 7.36328125 disk.rbd1.in‚0 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 438.671875 disk.rbd4.in‚0 disk.rbd1.out‚ 4.66259765625$ disk./.usageGB‚1098.768466949463 disk.sda3.out‚ 438.671875 disk.sda2.out‚ 0.0390625) proc.memory.percent‚17.46692657470703( disk./.usagePercent‚73.7392737451289g@Fø%‚(:„ ¸ìäÔ’¿¢x gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šz³Êtù%‚(:é ¸ìäÔù÷î{ disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 proc.memory.rssMB‚ 2861.78125& disk.nvme0n1.out‚42237.34130859375 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0$ memory_percent‚97.93798923492432 cpu‚25.277327679303248 proc.cpu.threads‚17) disk./.usagePercent‚73.73916110181383( proc.memory.availableMB‚ 337.83984375) proc.memory.percent‚17.46692657470703 network.recv‚ 3.25574e+06 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 439.26171875 disk.rbd0.in‚0 disk.sda3.out‚ 439.22265625 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚42237.34130859375 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚42237.34130859375 disk.sda3.in‚7.3125 disk.dm-1.in‚7.3125 disk.dm-1.out‚ 439.22265625 disk.drbd1777.in‚ 1.03515625$ disk./.usageGB‚1098.766788482666 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.36328125! disk.rbd2.out‚266.37255859375 disk.rbd1.out‚ 4.66259765625 network.sent‚ 2.3865884e+07¯\Mú%‚(:… ÇìäÔõ¼šx gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚Q°žoû%‚(:ä ÇìäÔ·¿¿z disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚7.3125 disk.rbd4.in‚0 disk.sda3.in‚7.3125( proc.memory.availableMB‚ 337.83984375 disk.rbd3.out‚0 network.recv‚ 3.260362e+06 disk.dm-1.out‚ 439.84375! disk.rbd2.out‚266.37255859375 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 proc.cpu.threads‚17) disk./.usagePercent‚73.73916340588164 disk.rbd1.out‚ 4.66259765625& disk.nvme0n1.out‚42237.57568359375 disk.rbd3.in‚0) proc.memory.percent‚17.46692657470703 network.sent‚ 2.3871899e+07 disk.drbd1777.out‚0 disk.rbd4.out‚0( disk.nvme0c0n1.out‚42237.57568359375 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda3.out‚ 439.84375$ memory_percent‚97.93798923492432 disk.dm-0.in‚5.3125 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125% disk./.usageGB‚1098.7668228149414# disk.dm-0.out‚42237.57568359375 disk.rbd0.in‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 439.8828125 proc.memory.rssMB‚ 2861.78125 cpu‚25.276968604461675u€×2ü%- _wandbruntime‚9272  _runtime‚9272bƒXÇý%Á trainer/global_step‚9599 % _timestamp‚1.7884258018029966e+09 _runtime‚9272.552961324 _step‚785 & lr-AdamW/pg1‚9.935616759030288e-05 & lr-AdamW/pg2‚9.935616759030288e-05ÞÓÅ'Ìþ%Æ % _timestamp‚1.7884258018029966e+09 _runtime‚9272.552961324 _step‚785 & lr-AdamW/pg1‚9.935616759030288e-05 & lr-AdamW/pg2‚9.935616759030288e-05 trainer/global_step‚9599‘÷N]2ÿ%- _wandbruntime‚9272  _runtime‚9272xUî­ï'€&éO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407333588259840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18126794745344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚377198018685630 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17816869092864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 426311317 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 _step‚786 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 trainer/global_step‚9599 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 176701329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚377198259374138 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379589398351872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397406267444736 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167880197 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚379589416354816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884258018037167e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397406513651200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397406267444736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17816869092864 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚359432849476608 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 249609986 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚425460406945280 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 176701331 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚379589398351872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17765169209022 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103372396 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 249584316S˜ô'&îO UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 176701331 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 167879566 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚9272.553830041 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚425460116280832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 176701329 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 249609986 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64507172 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚359432849476608 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 249609986 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚425460406945280 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 176701331 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚379589398351872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17765169209022 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103372396 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 249584316 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407333588259840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18126794745344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚377198018685630 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17816869092864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 426311317 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 _step‚786 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 trainer/global_step‚9599 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 176701329 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚377198259374138 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379589398351872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397406267444736 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167880197 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚379589416354816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884258018037167e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397406513651200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397406267444736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17816869092864 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 249609867 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 SEDevîÔy®iceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 249635563 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64513680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚377236394231802 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚379627754298880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚379627754298880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379627736295936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17818706787328 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176719050 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 trainer/global_step‚9599 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 426328915 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103382751 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359469168973824 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359469151934464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17767001608622 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18128668562432 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17818935004160 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚397446689303040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 103382749 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚377236153543086 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚397446443083264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚787 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 249609867 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 426354613 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17767225257978 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167897106 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407374757956096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176719048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 426354613 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17818706787328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17818935004160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚379627736295936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167896429 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800² þeþ&„&øM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 249609867 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 249635563 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64513680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚377236394231802 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚379627754298880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚379627754298880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379627736295936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17818706787328 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176719050 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 trainer/global_step‚9599 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 426328915 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103382751 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359469168973824 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359469151934464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17767001608622 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18128668562432 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17818935004160 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚397446689303040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 103382749 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚377236153543086 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚397446443083264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚787 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 249609867 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 426354613 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17767225257978 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167897106 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407374757956096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176719048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 426354613 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17818706787328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17818935004160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚379627736295936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167896429 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64514355 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚425503145127424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176719050 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 426328915 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407374781896192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18128387171328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360“=å2…&- _wandbruntime‚9273  _runtime‚9273³iåË\†&Ö /loss/FRt3/detection_loss‚2.2585964131614436 0loss/FRt3/assignment_loss‚2.5799390608849735 2loss/SRqqt2/assignment_loss‚0.8045825652721653 1loss/SRqqt4/detection_loss‚1.8445575319706125 epoch‚7 _runtime‚9273.415308398 .loss/FRt1/assignment_loss‚2.96911747862653 .loss/FBt1/detection_loss‚2.326283276081085 /loss/FBt2/detection_loss‚0.5628943691738475 # _timestamp‚1.78842580266548e+09 _step‚788 1loss/FBt3/assignment_loss‚0.22221785032888874 1loss/FBt4/assignment_loss‚0.19743264205753802 /loss/FRt4/detection_loss‚2.3718067793469553 1loss/SRqqt1/detection_loss‚2.2782653535113617 .loss/FBt4/detection_loss‚3.704076278209686 2loss/SRqqt1/assignment_loss‚1.4375377429430098 2loss/SRqqt4/assignment_loss‚1.7190831007158502 /loss/FRt1/detection_loss‚2.4392358987796596 trainer/global_step‚9599 1loss/SRqqt2/detection_loss‚0.7481275422991637 1loss/SRqqt3/detection_loss‚1.6655962809043772 0loss/FRt4/assignment_loss‚2.8235602487217295 2loss/SRqqt3/assignment_loss‚1.3871225535997427 0loss/FBt1/assignment_loss‚0.2291373760080104 1loss/FBt2/assignment_loss‚0.02490116772157826 /loss/FRt2/detection_loss‚1.4525547949772961 /loss/FBt3/detection_loss‚1.7522523005803425 %loss/total_loss‚39.44080754744958 0loss/FRt2/assignment_loss‚1.6419289415737368à÷pda‡&Û 0loss/FRt3/assignment_loss‚2.5799390608849735 2loss/SRqqt2/assignment_loss‚0.8045825652721653 1loss/SRqqt4/detection_loss‚1.8445575319706125 epoch‚7 _runtime‚9273.415308398 .loss/FRt1/assignment_loss‚2.96911747862653 .loss/FBt1/detection_loss‚2.326283276081085 /loss/FBt2/detection_loss‚0.5628943691738475 # _timestamp‚1.78842580266548e+09 _step‚788 1loss/FBt3/assignment_loss‚0.22221785032888874 1loss/FBt4/assignment_loss‚0.19743264205753802 /loss/FRt4/detection_loss‚2.3718067793469553 1loss/SRqqt1/detection_loss‚2.2782653535113617 .loss/FBt4/detection_loss‚3.704076278209686 2loss/SRqqt1/assignment_loss‚1.4375377429430098 2loss/SRqqt4/assignment_loss‚1.7190831007158502 /loss/FRt1/detection_loss‚2.4392358987796596 trainer/global_step‚9599 1loss/SRqqt2/detection_loss‚0.7481275422991637 1loss/SRqqt3/detection_loss‚1.6655962809043772 0loss/FRt4/assignment_loss‚2.8235602487217295 2loss/SRqqt3/assignment_loss‚1.3871225535997427 0loss/FBt1/assignment_loss‚0.2291373760080104 1loss/FBt2/assignment_loss‚0.02490116772157826 /loss/FRt2/detection_loss‚1.4525547949772961 /loss/FBt3/detection_loss‚1.7522523005803425 %loss/total_loss‚39.44080754744958 0loss/FRt2/assignment_loss‚1.6419289415737368 /loss/FRt3/detection_loss‚2.2585964131614436”•r5±ˆ&‚(:… ÖìäÔÄí¶x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.194' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8388 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰Æ‚]l‰&‚(:á ÖìäÔ®¡‚{ disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.rbd1.in‚0# disk.dm-0.out‚42337.11962890625 proc.memory.rssMB‚ 2861.78125 cpu‚25.27657315945941 network.sent‚ 2.3877764e+07 disk.drbd1777.out‚0 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda3.in‚7.3125) proc.memory.percent‚17.46692657470703 network.recv‚ 3.264455e+06 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.rbd2.out‚ 266.482421875 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0& disk.nvme0n1.out‚42337.11962890625( proc.memory.availableMB‚ 337.73046875( disk.nvme0c0n1.out‚42337.11962890625 disk.rbd3.in‚0 disk.sda.out‚440.5625 disk.sda3.out‚ 440.5234375) disk./.usagePercent‚73.73928859356587 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 440.5234375 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125# memory_percent‚97.9386568069458 proc.cpu.threads‚17 disk.rbd1.out‚ 4.66259765625% disk./.usageGB‚1098.7686882019043‘ÐöŠ&‚(:… åìäÔÒ¦¯x gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ÚõÃo‹&‚(:ä åìäÔˆÏÖz disk.sda3.in‚7.3125( proc.memory.availableMB‚ 337.84765625 proc.memory.rssMB‚ 2861.78125 proc.cpu.threads‚17 network.sent‚ 2.3986729e+07 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 266.482421875 disk.rbd4.out‚0 disk.sda3.out‚ 441.0078125& disk.nvme0n1.out‚42436.16259765625 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0) disk./.usagePercent‚73.73928859356587% disk./.usageGB‚1098.7686882019043# disk.dm-0.out‚42436.16259765625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd1.in‚0# memory_percent‚97.9379415512085) proc.memory.percent‚17.46692657470703 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 441.0078125 disk.rbd0.out‚0 disk.nvme0n1.in‚5.3125 disk.rbd3.in‚0 disk.dm-1.in‚7.3125 network.recv‚ 3.272058e+06 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚42436.16259765625 disk.rbd1.out‚ 4.66259765625 cpu‚25.276183879687967 disk.sda2.in‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 441.046875*Ô…Œ&‚(:… ôìäԸƸx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.896' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ŠL!p&‚(:å ôìäÔæË‚{) proc.memory.percent‚17.46692657470703 disk.dm-1.in‚7.3125% disk./.usageGB‚1098.7705307006836 disk.rbd2.out‚ 268.193359375 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda3.in‚7.3125% disk.nvme0n1.out‚42537.8017578125) disk./.usagePercent‚73.73941224520492 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.36328125 disk.sda.out‚ 441.42578125' proc.memory.availableMB‚ 338.0859375 cpu‚25.275818750916553 network.sent‚ 2.3992149e+07 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 441.38671875" disk.dm-0.out‚42537.8017578125 disk.rbd2.in‚0$ memory_percent‚97.93648719787598 proc.cpu.threads‚17 network.recv‚ 3.275595e+06 disk.dm-1.out‚ 441.38671875 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0' disk.nvme0c0n1.out‚42537.8017578125 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 4.66259765625 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 proc.memory.rssMB‚ 2861.78125 |t2Ž&- _wandbruntime‚9317  _runtime‚9317bêOÆ&À trainer/global_step‚9649 $ _timestamp‚1.788425847244867e+09 _runtime‚9317.994790241 _step‚789 & lr-AdamW/pg1‚9.557257189971283e-05 & lr-AdamW/pg2‚9.557257189971283e-054¾Ë&Å & lr-AdamW/pg1‚9.557257189971283e-05 & lr-AdamW/pg2‚9.557257189971283e-05 trainer/global_step‚9649 $ _timestamp‚1.788425847244867e+09 _runtime‚9317.994790241 _step‚789•8 F'2‘&- _wandbruntime‚9317  _runtime‚9317cm5s’&çO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17908996433408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 # _timestamp‚1.78842584724569e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64835638 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 428452497 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103890146 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103890144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379104775149638 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚361247989408768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 250865218 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18219960589824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚427612330618368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 250890888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 177587281 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚427612033662464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409392097012736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚361247972369408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17908768229888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64834986 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17908996433408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 _step‚790 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚654 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399415309991424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚381506313558016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 250890888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379105015838146 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17908768229888 _runtime‚9317.995779435 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399415063784960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚381506295555072 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 177587281 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 177587279 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17856802780230 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 428478169 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18220233605632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17857026429378 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.cròn"è urrent‚ 273015808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168725784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚381506295555072 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399415309991424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 trainer/global_step‚9649 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚381506313558016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 250865218 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18220233605632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17857026429378 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168725784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚381506295555072 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399415309991424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 trainer/global_step‚9649 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚381506313558016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 250865218 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 _step‚790 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚654 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399415309991424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚381506313558016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 250890888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379105015838146 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17908768229888 _runtime‚9317.995779435 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399415063784960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚381506295555072 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 177587281 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 177587279 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17856802780230–øÍ2”&- _wandbruntime‚9318  _runtime‚9318øñJø&•&òM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17910834141184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379143150695810 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚381544633499136 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 250916465 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168741724 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379142910007094 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 428495767 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17910605924352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18221826637824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17910605924352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 428521465 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64841225 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚361284274827264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 168742392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 trainer/global_step‚9649 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚427655069406720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 177604998 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409433266708992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 177605000 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚381544651502080 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17858858829186 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚361284291866624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚791 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399455485643264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103900501 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18222099640320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _runtime‚9318.867880452 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚381544651502080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 250890769 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 250890769 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17910834141184 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 250916465 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379143150695810 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚381544633499136 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 250916465 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168741724 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379142910007094 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 428495767 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17910605924352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18221826637824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17910605924352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 428521465 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64841225 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚361284274827264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 168742392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 trainer/global_step‚9649 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚427655069406720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 177604998 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409433266708992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 177605000 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚381544651502080 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17858858829186 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚361284291866624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚791 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3994554856zæfö$43264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103900501 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18222099640320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _runtime‚9318.867880452 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚381544651502080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 250890769—`T^2—&- _wandbruntime‚9318  _runtime‚9318zv™øY˜&Ó 0loss/FRt1/assignment_loss‚3.2645189122455878 2loss/SRqqt2/assignment_loss‚0.8889246905934723 2loss/SRqqt3/assignment_loss‚1.2372992214347636 0loss/FBt1/assignment_loss‚0.3943346667098012 .loss/FRt4/detection_loss‚2.193129783256272 .loss/FBt4/detection_loss‚2.112242204802377 trainer/global_step‚9649 $ _timestamp‚1.788425848119065e+09 _step‚792 2loss/SRqqt4/assignment_loss‚1.4712372165545822 _runtime‚9318.868870608 0loss/FRt2/assignment_loss‚1.5391652250289918 2loss/SRqqt1/assignment_loss‚1.2372702106833458 /loss/FRt1/detection_loss‚2.4865270587729245 0loss/SRqqt1/detection_loss‚2.426372225085894 0loss/SRqqt3/detection_loss‚1.906712949884181 1loss/SRqqt4/detection_loss‚1.7384138506650926 0loss/FRt3/assignment_loss‚3.1126138969607977 .loss/FRt3/detection_loss‚2.323281028348467 /loss/FRt4/assignment_loss‚3.562532940000858 1loss/FBt2/assignment_loss‚0.16517055831801136 /loss/FBt2/detection_loss‚0.4016627173592476 .loss/FBt3/detection_loss‚5.237366338570912 %loss/total_loss‚43.27116579786647 1loss/FBt3/assignment_loss‚0.05314983303348223 0loss/FBt4/assignment_loss‚0.5636915729846805 /loss/FRt2/detection_loss‚1.6028372719138861 1loss/SRqqt2/detection_loss‚0.8931182418566417 /loss/FBt1/detection_loss‚2.4595931828022004 epoch‚7€—Z^™&Ø 0loss/FRt2/assignment_loss‚1.5391652250289918 2loss/SRqqt1/assignment_loss‚1.2372702106833458 /loss/FRt1/detection_loss‚2.4865270587729245 0loss/SRqqt1/detection_loss‚2.426372225085894 0loss/SRqqt3/detection_loss‚1.906712949884181 1loss/SRqqt4/detection_loss‚1.7384138506650926 0loss/FRt3/assignment_loss‚3.1126138969607977 .loss/FRt3/detection_loss‚2.323281028348467 /loss/FRt4/assignment_loss‚3.562532940000858 1loss/FBt2/assignment_loss‚0.16517055831801136 /loss/FBt2/detection_loss‚0.4016627173592476 .loss/FBt3/detection_loss‚5.237366338570912 %loss/total_loss‚43.27116579786647 1loss/FBt3/assignment_loss‚0.05314983303348223 0loss/FBt4/assignment_loss‚0.5636915729846805 /loss/FRt2/detection_loss‚1.6028372719138861 1loss/SRqqt2/detection_loss‚0.8931182418566417 /loss/FBt1/detection_loss‚2.4595931828022004 epoch‚7 0loss/FRt1/assignment_loss‚3.2645189122455878 2loss/SRqqt2/assignment_loss‚0.8889246905934723 2loss/SRqqt3/assignment_loss‚1.2372992214347636 0loss/FBt1/assignment_loss‚0.3943346667098012 .loss/FRt4/detection_loss‚2.193129783256272 .loss/FBt4/detection_loss‚2.112242204802377 trainer/global_step‚9649 $ _timestamp‚1.788425848119065e+09 _step‚792 2loss/SRqqt4/assignment_loss‚1.4712372165545822 _runtime‚9318.868870608˜¶RW•š&‚(:… ƒíäÔ¿•®x gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.792' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á+ip›&‚(:å ƒíäÔËò‚{ disk.sda3.in‚7.3125 disk.nvme0n1.in‚5.3125 disk.dm-1.out‚ 442.05859375 disk.drbd1777.in‚ 1.03515625 network.recv‚ 3.279937e+06 disk.sda3.out‚ 442.05859375 disk.rbd2.out‚ 268.201171875 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.rbd3.in‚0 network.sent‚ 2.3998159e+07' disk.nvme0c0n1.out‚42636.9150390625 disk.rbd3.out‚0 disk.sda.out‚ 442.09765625 cpu‚25.27548342864435 proc.cpu.threads‚17% disk./.usageGB‚1098.7705307006836 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.sda1.in‚ 0.05078125$ memory_percent‚97.93832302093506( proc.memory.availableMB‚ 337.78515625) proc.memory.percent‚17.46692657470703) disk./.usagePercent‚73.73941224520492 disk.dm-1.in‚7.3125 disk.rbd2.in‚0 disk.rbd4.in‚0 proc.memory.rssMB‚ 2861.78125 disk.rbd1.in‚0 disk.rbd1.out‚ 4.66259765625 disk.dm-0.in‚5.3125" disk.dm-0.out‚42636.9150390625 disk.sda1.out‚0 disk.sda.in‚ 7.36328125% disk.nvme0n1.out‚42636.9150390625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125“˜Ìœ&‚(:„ ’íäÔäóx gpu.0.gpu‚41.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05:™ts&‚(:è ’íäÔç÷Ì{ disk.rbd3.out‚0 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.in‚ 7.36328125$ memory_percent‚97.93705940246582 cpu‚25.277947076950195 network.sent‚ 2.4105821e+07 network.recv‚ 3.286129e+06% disk./.usageGB‚1098.7705307006836 disk.rbd1.out‚ 4.66259765625 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0) disk./.usagePercent‚73.73941224520492 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda3.in‚7.3125 disk.dm-0.in‚5.3125 disk.rbd0.in‚0' proc.memory.availableMB‚ 337.9921875# disk.dm-0.out‚42637.77783203125 disk.dm-1.in‚7.3125 disk.rbd0.out‚0 disk.rbd3.in‚0) proc.memory.percent‚17.46692657470703& disk.nvme0n1.out‚42637.77783203125( disk.nvme0c0n1.out‚42637.77783203125 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.dm-1.out‚ 442.56640625 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 442.60546875 proc.memory.rssMB‚ 2861.78125 disk.sda3.out‚ 442.56640625 disk.rbd2.out‚ 268.279296875BåûjŽž&‚(:ƒ ¡íäÔ°ñºx gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à)þPmŸ&‚(:â ¡íäÔÜ€Êz& disk./.usagePercent‚73.73941275722 disk.rbd2.out‚ 268.279296875 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.66259765625 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 442.98828125 disk.rbd4.out‚0' disk.nvme0c0n1.out‚42736.8974609375( proc.memory.availableMB‚ 337.84765625 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703 cpu‚25.284912861602002 network.sent‚ 2.4112157e+07% disk./.usageGB‚1098.7705383300781 disk.sda1.out‚0 disk.sda3.in‚7.3125 disk.sda3.out‚ 442.98828125" disk.dm-0.out‚42736.8974609375 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0 disk.sda.in‚ 7.36328125 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0# memory_percent‚97.9379415512085 proc.cpu.threads‚17 disk.nvme0n1.in‚5.3125% disk.nvme0n1.out‚42736.8974609375 network.recv‚ 3.290966e+06 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 443.02734375 disk.sda2.out‚ 0.0390625‚ÄYí &‚(:„ °íäÔåÉÉx gpu.0.gpu‚44.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(vÌo¡&‚(:ä °íäÔ¸ýz network.sent‚ 2.411743e+07 disk.sda2.out‚ 0.0390625 disk.sda3.out‚ 443.546875# disk.dm-0.out‚42836.19482421875 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚42836.19482421875 proc.cpu.threads‚17& disk.nvme0n1.out‚42836.19482421875 disk.dm-1.in‚7.3125! disk.rbd2.out‚268.46337890625 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0$ memory_percent‚97.93894290924072 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda.out‚ 443.5859375( proc.memory.availableMB‚ 337.68359375 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703) disk./.usagePercent‚73.73928987360355 disk.rbd1.in‚0 disk.rbd1.out‚ 4.66259765625 disk.rbd3.in‚0 disk.sda3.in‚7.3125 cpu‚25.29179529279275 network.recv‚ 3.294374e+06 disk.dm-0.in‚5.3125 disk.rbd0.out‚0 disk.sda.in‚ 7.36328125% disk./.usageGB‚1098.7687072753906 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.nvme0n1.in‚5.3125 disk.dm-1.out‚ 443.546875 disk.rbd4.out‚0¸•¹tš¢&‚(: ¿íäÔ†óÁx gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.526' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.705199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û ôs£&‚(:è ¿íäÔ«Óz disk.sda3.out‚ 444.11328125& disk.nvme0n1.out‚42935.23388671875# disk.dm-0.out‚42935.23388671875 disk.rbd2.in‚0( disk.nvme0c0n1.out‚42935.23388671875 disk.rbd1.out‚ 4.66259765625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125 network.sent‚ 2.4123442e+07 network.recv‚ 3.299054e+06 disk.sda.in‚ 7.36328125 disk.sda.out‚ 444.15234375 disk.dm-1.in‚7.3125$ memory_percent‚97.94125556945801 proc.cpu.threads‚17 disk.sda3.in‚7.3125 disk.sda2.out‚ 0.0390625 disk.nvme0n1.in‚5.3125 disk.drbd1777.in‚ 1.03515625 cpu‚25.2984475256241 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 disk.sda2.in‚0! disk.rbd2.out‚268.46337890625 disk.rbd4.out‚0' proc.memory.availableMB‚ 337.3046875 proc.memory.rssMB‚ 2861.78125) proc.memory.percent‚17.46692657470703) disk./.usagePercent‚73.73927707322684% disk./.usageGB‚1098.7685165405273 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 444.11328125 disk.drbd1777.out‚0 disk.rbd4.in‚0üL7 2¤&- _wandbruntime‚9391  _runtime‚9391°À4EV*¥&ÐT XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337968128 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚561 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 537294673 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 430590702 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 171710976 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 430590702 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 184860760 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚561 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 251884223 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6739344896 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1261 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 104567547 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚262 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 178726680 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚381214199284766 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20203 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚17970357011968 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 251884223 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 169815799 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚18282515446784 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚17918358832158 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 6893338624 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6739344896 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 6893338624 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 549453824 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚299 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚262 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚383668966966784 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚373 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 173093376 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31397 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31168 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚23 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚383668984969728 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 149153280 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚561 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 169815426 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚299 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 189713920 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 104567545 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚411682310688256 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 430610905 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚65247881 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚18282366293504 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚299 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚401639323978752 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20203 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20201 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚429964850075136 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚561 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 178726680 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ^QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚90898432 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚363295840452608 _runtime‚9391.313875694 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚383668966966784 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚371 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚363295823413248 _step‚793 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚383668984969728 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 251864022 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚65248252 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6252512256 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 178726682 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚401639513692672 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 178726682 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 trainer/global_step‚9671 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 251864022 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚401639323978752 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚17970528722944 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31168 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚23 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚383668984969728 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 149153280 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚561 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 169815426 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚299 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 189713920 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 104567545 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚411682310688256 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 430610905 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚65247881 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚18282366293504 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚299 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚401639323978752 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20203 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20201 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚429964850075136 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚561 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 178726680 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ^QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚90898432 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚363295840452608 _runtime‚9391.313875694 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚383668966966784 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚371 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚363295823413248 _step‚793 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚383668984969728 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 251864022 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚65248252 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6252512256 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 178726682 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚401639513692672 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 178726682 B;DeviceStatsMonitor.on_validation_batch_start/max_split_sizeÖz©Ío‚-1 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 trainer/global_step‚9671™ø'v2§&- _wandbruntime‚9392  _runtime‚9392ÇÂX‡`)¨&ÚR RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6739344896 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚17918582187654 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31168 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 178741618 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 167821400 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 430613914 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31168 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚65253277 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31397 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚17970921518080 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 541892608 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 549453824 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 251892499 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 6893338624 \ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚90898432 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚383695107756544 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚299 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20201 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚381239158550150 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚299 % _timestamp‚1.7884259216233275e+09 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚401665857563648 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 430634117 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚401666047277568 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚401666047277568 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31397 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚363320593401856 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 178741616 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 537294673 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 6893338624 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 6893338624 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 430634117 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚401666047277568 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚401666047277568 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31397 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚363320593401856 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 178741616 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 537294673 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 6893338624 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 6893338624 B&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14246712625026703 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12763918936252594 particle/accuracy_5_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5066666603088379 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8164771795272827 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20879369974136353 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9927744269371033 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17361685633659363 particle/accuracy_3_of_4‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43924668431282043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4563405215740204 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 epoch‚7 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39854419231414795 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4402356445789337 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1508457064628601 +particle/specificity‚0.9491645693778992 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4907570481300354 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007709750905632973 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.43843841552734375 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_4‚0.9059877991676331 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13559217751026154 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16637426614761353 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5362867712974548 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.43896329402923584 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4145223796367645 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975993633270264 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841194152832 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9952381253242493 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.7777777910232544 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.021285492926836014 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.03947368264198303 +jet/accuracy_3_of_4‚0.06806861609220505 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9259259104728699 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9898778200149536 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.696969747543335 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17139367759227753 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08416985720396042 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2291666567325592 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5315988659858704 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5373626947402954 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.021ÂõW†m14335261285305 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25248560309410095 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02609211951494217 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14263290166854858 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.008862434886395931 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.39544007182121277 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9965599179267883 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.16717171669006348 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 *loss/val_total_loss‚41.465494945986116 particle/accuracy_2_of_4‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5217974781990051 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 -jet/accuracy_4_of_5‚0.0073220692574977875 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16054749488830566 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20359043776988983 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35880032181739807 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12924399971961975 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4721716046333313 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8510561585426331 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09555555135011673 jet/accuracy_6_of_6‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976491391658783 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.08197370916604996 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.519132673740387 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666567325592 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2490740716457367 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4721716046333313 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11958636343479156 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21925286948680878 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 *jet/accuracy_2_of_5‚0.4776689410209656 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_6‚1 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.399673193693161 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875277280807495 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.42763158679008484 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 particle/accuracy_1_of_7‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 *jet/accuracy_1_of_1‚0.4698995053768158 particle/accuracy_2_of_5‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36944445967674255 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5076947212219238 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0023969856556504965 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018891867250204086 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2460317611694336 *jet/accuracy_1_of_2‚0.7404640316963196 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 _step‚795 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12310855090618134 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3218535780906677 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4877684414386749 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.180632084608078 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007075651548802853 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09071240574121475 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.4022018015384674 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1724541038274765 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_1‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.24389007687568665 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15147565305233002 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01937984675168991 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 jet/accuracy_1_of_7‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.46814364194869995 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4747304320335388 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36996254324913025 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 jet/accuracy_2_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9852662086486816 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.985467791557312 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5560259819030762 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.46752792596817017 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.08812260627746582 *jet/accuracy_2_of_6‚0.3525640666484833 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14135092496871948 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3857584297657013 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9917725324630737 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3132440447807312 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18511120975017548 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.43672841787338257 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964095950126648 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 particle/accuracy_1_of_6‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4187205731868744 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5219876170158386 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.175615131855011 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14373953640460968 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 +particle/sensitivity‚0.3210391104221344 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 jet/accuracy_7_of_7‚0 % _timestamp‚1.7884259356572125e+09 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.46115100383758545 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24093510210514069 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_4‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4197530746459961 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 particle/accuracy_1_of_5‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.33707740902900696 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5036028623580933 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9908026456832886 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2191171646118164 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10241428017616272 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.49703383445739746 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.988304078578949 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16824324429035187 ,jet/accuracy_3_of_3‚0.019036831334233284 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.48232755064964294 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1611230969429016 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.21666666865348816 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.39945533871650696 particle/accuracy_6_of_7‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.32653722167015076 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4084290862083435 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.413194477558136 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01668505184352398 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1611230969429016 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15728141367435455 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.052463945001363754 +jet/accuracy_2_of_3‚0.23397932946681976 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6429006457328796 *jet/accuracy_1_of_5‚0.9394137263298035 particle/accuracy_1_of_4‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 particle/accuracy_3_of_7‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.989120602607727 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.16435186564922333 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4763514995574951 5 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.98167884349823 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38737237453460693 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4238350987434387 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06586581468582153 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.250615656375885 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04238505661487579 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03193804621696472 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43395721912384033 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 jet/accuracy_5_of_7‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1374434381723404 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14661148190498352 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.073117695748806 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5275033116340637 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 jet/accuracy_4_of_6‚0 jet/accuracy_3_of_7‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10040431469678879 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1505747139453888 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4501389265060425 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5416666865348816 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4850749671459198 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5760416388511658 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48222213983535767 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9906619787216187 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02444610744714737 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.489470899105072 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 +jet/accuracy_3_of_6‚0.13461539149284363 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.7777777910232544 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.48201626539230347 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9285714030265808 particle/accuracy_2_of_7‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9712643623352051 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 _runtime‚9406.409950588 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23685063421726227 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09219857305288315 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994708895683289 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38876521587371826 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.014330637641251087 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2888889014720917 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2497742921113968 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5073186755180359 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.250935435295105 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.020256992429494858 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3436535894870758 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚0.5 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.248212531208992 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.582684338092804 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40602681040763855 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_6‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9954648613929749 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3778461515903473 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.989004909992218 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4698079526424408 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09259258955717087 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 trainer/global_step‚9671 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5086860060691833 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012948965653777122 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2068352997303009 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4528365135192871 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.48915398120880127 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28260359168052673 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19407543540000916 particle/accuracy_3_of_5‚1 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.46408048272132874 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754689216614 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 particle/accuracy_2_of_6‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6701586842536926 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.42846229672431946 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.09899797290563583 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5412442088127136 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.49045565724372864 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5073739886283875 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06239447742700577 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5570970773696899 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1410416066646576 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.45127609372138977 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18802683055400848 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1628762185573578 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.504429817199707 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5408052206039429 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9989040493965149 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.004917993675917387 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2659654915332794 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5362622737884521 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5317906737327576 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14438800513744354 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0=Éöl®&°§ ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 particle/accuracy_1_of_5‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.33707740902900696 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5036028623580933 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9908026456832886 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2191171646118164 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10241428017616272 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.49703383445739746 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.988304078578949 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16824324429035187 ,jet/accuracy_3_of_3‚0.019036831334233284 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.48232755064964294 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1611230969429016 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.21666666865348816 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.39945533871650696 particle/accuracy_6_of_7‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.32653722167015076 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4084290862083435 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.413194477558136 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01668505184352398 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1611230969429016 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15728141367435455 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.052463945001363754 +jet/accuracy_2_of_3‚0.23397932946681976 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6429006457328796 *jet/accuracy_1_of_5‚0.9394137263298035 particle/accuracy_1_of_4‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 particle/accuracy_3_of_7‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.989120602607727 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.16435186564922333 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4763514995574951 5 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.98167884349823 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38737237453460693 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4238350987434387 >&Purity/0FRt2SRqqt1FBt/event_proportion‚á«ù0.00936259888112545 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06586581468582153 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.250615656375885 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04238505661487579 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03193804621696472 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43395721912384033 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 jet/accuracy_5_of_7‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1374434381723404 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14661148190498352 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.073117695748806 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5275033116340637 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 jet/accuracy_4_of_6‚0 jet/accuracy_3_of_7‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10040431469678879 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1505747139453888 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4501389265060425 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5416666865348816 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4850749671459198 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5760416388511658 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48222213983535767 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9906619787216187 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02444610744714737 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.489470899105072 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 +jet/accuracy_3_of_6‚0.13461539149284363 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.7777777910232544 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.48201626539230347 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9285714030265808 particle/accuracy_2_of_7‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9712643623352051 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 _runtime‚9406.409950588 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23685063421726227 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09219857305288315 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994708895683289 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38876521587371826 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.014330637641251087 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2888889014720917 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2497742921113968 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5073186755180359 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.250935435295105 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.020256992429494858 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3436535894870758 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚0.5 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.248212531208992 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.582684338092804 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40602681040763855 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_6‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9954648613929749 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3778461515903473 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.989004909992218 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4698079526424408 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09259258955717087 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 trainer/global_step‚9671 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5086860060691833 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012948965653777122 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2068352997303009 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4528365135192871 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.48915398120880127 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28260359168052673 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19407543540000916 particle/accuracy_3_of_5‚1 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.46408048272132874 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754689216614 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 particle/accuracy_2_of_6‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6701586842536926 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.42846229672431946 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.09899797290563583 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5412442088127136 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.49045565724372864 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5073739886283875 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06239447742700577 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5570970773696899 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1410416066646576 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.45127609372138977 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18802683055400848 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1628762185573578 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.504429817199707 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5408052206039429 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9989040493965149 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.004917993675917387 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2659654915332794 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5362622737884521 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5317906737327576 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14438800513744354 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9916446208953857 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887130260467529 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 5validation_average_jet_accuracy‚0.423589825630188 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4233333170413971 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.45673054456710815 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03482142835855484 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5336309671401978 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9907397627830505 *jet/accuracy_1_of_3‚0.8571389317512512 +jet/accuracy_3_of_5‚0.07553122937679291 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5208333134651184 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9938086867332458 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4089147448539734 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14246712625026703 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12763918936252594 particle/accuracy_5_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5066666603088379 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8164771795272827 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20879369974136353 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9927744269371033 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17361685633659363 particle/accuracy_3_of_4‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43924668431282043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4563405215740204 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 epoch‚7 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39854419231414795 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4402356445789337 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1508457064628601 +particle/specificity‚0.9491645693778992 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4907570481300354 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007709750905632973 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.43843841552734375 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_4‚0.9059877991676331 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13559217751026154 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16637426614761353 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5362867712974548 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.43896329402923584 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4145223796367645 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975993633270264 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841194152832 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9952381253242493 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.7777777910232544 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.021285492926836014 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.03947368264198303 +jet/accuracy_3_of_4‚0.06806861609220505 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9259259104728699 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9898778200149536 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.696969747543335 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17139367759227753 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08416985720396042 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2291666567325592 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5315988659858704 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5373626947402954 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25248560309410095 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02609211951494217 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14263290166854858 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.008862434886395931 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.39544007182121277 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9965599179267883 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.16717171669006348 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 *loss/val_total_loss‚41.465494945986116 particle/accuracy_2_of_4‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5217974781990051 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 -jet/accuracy_4_of_5‚0.0073220692574977875 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16054749488830566 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20359043776988983 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35880032181739807 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12924399971961975 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4721716046333313 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8510561585426331 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09555555135011673 jet/accuracy_6_of_6‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976491391658783 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.08197370916604996 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.519132673740387 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666567325592 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2490740716457367 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4721716046333313 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11958636343479156 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21925286948680878 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 *jet/accuracy_2_of_5‚0.4776689410209656 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_6‚1 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.399673193693161 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875277280807495 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.42763158679008484 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 particle/accuracy_1_of_7‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 *jet/accuracy_1_of_1‚0.4698995053768158 particle/accuracy_2_of_5‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36944445967674255 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5076947212219238 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0023969856556504965 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018891867250204086 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2460317611694336 *jet/accuracy_1_of_2‚0.7404640316963196 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 _step‚795 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12310855090618134 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3218535780906677 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4877684414386749 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.180632084608078 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007075651548802853 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09071240574121475 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.4022018015384674 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1724541038274765 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_1‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.24389007687568665 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15147565305233002 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01937984675168991 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 jet/accuracy_1_of_7‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.46814364194869995 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4747304320335388 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36996254324913025 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 jet/accuracy_2_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9852662086486816 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.985467791557312 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5560259819030762 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.46752792596817017 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.08812260627746582 *jet/accuracy_2_of_6‚0.3525640666484833 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14135092496871948 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3857584297657013 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9917725324630737 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3132440447807312 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18511120975017548 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.43672841787338257 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964095950126648 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 particle/accuracy_1_of_6‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4187205731868744 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5219876170158386 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.175615131855011 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14373953640460968 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 +particle/sensitivity‚0.3210391104221344 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 jet/accuracy_7_of_7‚0 % _timestamp‚1.7884259356572125e+09 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.46115100383758545 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428À°¶iR312 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24093510210514069 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_4‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4197530746459961 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1›ÛÙÑV¯&‚(:… ÝíäÔ›·Çx gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.501' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.< _°&‚(:Ô ÝíäÔ¤¿Õ{ disk.sda.in‚ 7.36328125 disk.sda3.in‚7.3125 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.dm-0.out‚ 43201.53125 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.sda3.out‚ 445.1015625! proc.memory.rssMB‚ 2862.765625 disk.rbd1.out‚ 4.66259765625 disk.sda2.out‚ 0.0390625! disk.rbd2.out‚270.05322265625 disk.rbd3.in‚0 disk.sda.out‚ 445.140625 proc.cpu.threads‚17 network.sent‚ 2.4237983e+07 disk.dm-1.out‚ 445.1015625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0) disk./.usagePercent‚73.73943298181518 disk.dm-0.in‚5.3125 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0" disk.nvme0c0n1.out‚ 43201.53125' proc.memory.availableMB‚ 503.9453125 cpu‚25.305645255281057$ disk./.usageGB‚1098.770839691162$ memory_percent‚96.92416191101074) proc.memory.percent‚17.47293472290039 disk.nvme0n1.out‚ 43201.53125 disk.dm-1.in‚7.3125 disk.sda1.out‚0 network.recv‚ 3.310885e+06ÉåŒM2±&- _wandbruntime‚9434  _runtime‚9434O‚îMƲ&À & lr-AdamW/pg1‚9.185266003263864e-05 & lr-AdamW/pg2‚9.185266003263864e-05 trainer/global_step‚9699 $ _timestamp‚1.788425963403396e+09 _runtime‚9434.153349813 _step‚796 Ê·—˳&Å $ _timestamp‚1.788425963403396e+09 _runtime‚9434.153349813 _step‚796 & lr-AdamW/pg1‚9.185266003263864e-05 & lr-AdamW/pg2‚9.185266003263864e-05 trainer/global_step‚9699œU|'’2´&- _wandbruntime‚9434  _runtime‚9434ë[2ï'µ&éO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 252695115 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚364622548104192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚364622531064832 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚385070080516096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65498040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65497413 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 104972628 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431516467429888 _step‚797 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403095492964352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18338741116416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403095739170816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 179414197 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403095492964352 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170470668 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104972626 % _timestamp‚1.7884259634041646e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403095739170816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 252669445 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚413177702373376 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17973097603264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18025430451200 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18025430451200 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 179414199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18025658654720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18338470197760 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚413177702373376 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17973097603264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18025430451200 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18025430451200 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 179414199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18025658654720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18338470197760 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 179414197 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403095492964352 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170470668 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104972626 % _timestamp‚1.7884259634041646e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403095739170816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0œ¯Ðc™·&‚(:Ž ìíäÔÔü¹x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.44' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.887999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[Þ2³t¸&‚(:é ìíäÔ“©ñz' disk.nvme0c0n1.out‚43325.3330078125 disk.rbd3.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 445.81640625 disk.sda3.out‚ 445.77734375 disk.rbd1.in‚0 disk.rbd1.out‚ 4.66259765625) proc.memory.percent‚17.47293472290039 disk.dm-0.in‚5.3125 disk.dm-1.in‚7.3125 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0$ memory_percent‚96.92208766937256( proc.memory.availableMB‚ 504.28515625 proc.cpu.threads‚17 network.sent‚ 2.4361697e+07% disk.nvme0n1.out‚43325.3330078125" disk.dm-0.out‚43325.3330078125 disk.sda2.in‚0 disk.dm-1.out‚ 445.77734375 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda3.in‚7.3125) disk./.usagePercent‚73.73939227661724% disk./.usageGB‚1098.7702331542969 disk.rbd0.out‚0 cpu‚25.305244313034258 disk.rbd4.out‚0 disk.sda.in‚ 7.36328125! proc.memory.rssMB‚ 2862.765625 disk.rbd3.out‚0 network.recv‚ 3.317693e+06 disk.nvme0n1.in‚5.3125! disk.rbd2.out‚270.13134765625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125ƒ þ2¹&- _wandbruntime‚9435  _runtime‚9435²«ÊM º&óM trainer/global_step‚9699 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 % _timestamp‚1.7884259642581134e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 252720692 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 179431916 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170486734 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 307428352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18027268145664 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 432126912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18027496362496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 252720692 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65503753 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18027268145664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 432152610 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 179431918 _runtime‚9435.008629328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403135914822656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚382634004214268 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚798 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403135668602880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚385108400457216 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104982981 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17975153652220 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 XIDeviceStatsMo[q_¦¬nitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 179431916 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚385108400457216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65504379 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚413218872069632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚382633763525552 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0© áþ&»&øM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65504379 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚413218872069632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚382633763525552 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚9699 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 % _timestamp‚1.7884259642581134e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 252720692 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 179431916 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170486734 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 307428352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18027268145664 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 432126912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18027496362496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 252720692 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65503753 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18027268145664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 432152610 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 179431918 _runtime‚9435.008629328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403135914822656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚382634004214268 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚798 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403135668602880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚385108400457216 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104982981 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17975153652220 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 179431916 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚385108400457216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ž› !2¼&- _wandbruntime‚9435  _runtime‚9435Â϶n\½&Ö 2loss/SRqqt3/assignment_loss‚1.5832894622552685 0loss/FBt3/assignment_loss‚0.2752937659388408 /loss/FRt1/detection_loss‚2.4119216336144342 1loss/SRqqt2/detection_loss‚0.7719058622397593 0loss/SRqqt4/detection_loss‚1.635182070530067 0loss/FRt1/assignment_loss‚2.9937081336975098 /loss/FRt4/assignment_loss‚2.515315538307406 0loss/FBt1/assignment_loss‚0.9752780811312343 2loss/FBt2/assignment_loss‚0.060343953744989326 1loss/SRqqt3/detection_loss‚2.1376494547215903 % _timestamp‚1.7884259642599084e+09 0loss/FRt3/assignment_loss‚3.8174836568112642 /loss/FRt3/detection_loss‚2.2214578485713816 /loss/FRt4/detection_loss‚2.2497486387783625 trainer/global_step‚9699 1loss/SRqqt1/detection_loss‚2.1447789553891528 1loss/SRqqt1/assignment_loss‚1.524166361217133 2loss/SRqqt2/assignment_loss‚0.7503627843063658 1loss/SRqqt4/assignment_loss‚1.409394353565018 .loss/FBt4/detection_loss‚4.288783699274063 %loss/total_loss‚41.65559344729921 epoch‚8 0loss/FRt2/assignment_loss‚1.5005046243851001 /loss/FRt2/detection_loss‚1.6474043306393118 /loss/FBt1/detection_loss‚1.9432072469166346 _runtime‚9435.009709342 1loss/FBt4/assignment_loss‚0.27080343989655375 .loss/FBt2/detection_loss‚0.608806090471758 /loss/FBt3/detection_loss‚1.9188034608960152 _step‚799Rš*ºa¾&Û 1loss/SRqqt1/detection_loss‚2.1447789553891528 1loss/SRqqt1/assignment_loss‚1.524166361217133 2loss/SRqqt2/assignment_loss‚0.7503627843063658 1loss/SRqqt4/assignment_loss‚1.409394353565018 .loss/FBt4/detection_loss‚4.288783699274063 %loss/total_loss‚41.65559344729921 epoch‚8 0loss/FRt2/assignment_loss‚1.5005046243851001 /loss/FRt2/detection_loss‚1.6474043306393118 /loss/FBt1/detection_loss‚1.9432072469166346 _runtime‚9435.009709342 1loss/FBt4/assignment_loss‚0.27080343989655375 .loss/FBt2/detection_loss‚0.608806090471758 /loss/FBt3/detection_loss‚1.9188034608960152 _step‚799 2loss/SRqqt3/assignment_loss‚1.5832894622552685 0loss/FBt3/assignment_loss‚0.2752937659388408 /loss/FRt1/detection_loss‚2.4119216336144342 1loss/SRqqt2/detection_loss‚0.7719058622397593 0loss/SRqqt4/detection_loss‚1.635182070530067 0loss/FRt1/assignment_loss‚2.9937081336975098 /loss/FRt4/assignment_loss‚2.515315538307406 0loss/FBt1/assignment_loss‚0.9752780811312343 2loss/FBt2/assignment_loss‚0.060343953744989326 1loss/SRqqt3/detection_loss‚2.1376494547215903 % _timestamp‚1.7884259642599084e+09 0loss/FRt3/assignment_loss‚3.8174836568112642 /loss/FRt3/detection_loss‚2.2214578485713816 /loss/FRt4/detection_loss‚2.2497486387783625 trainer/global_step‚9699Ÿ}¼ìH¿&‚(:„ ûíäÔœ©Ÿy gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.503' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉZ÷qÀ&‚(:æ ûíäÔ¸ô¥z disk.rbd4.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚43567.02197265625 network.recv‚ 3.324713e+06% disk./.usageGB‚1098.7703170776367 disk.sda1.in‚ 0.05078125! proc.memory.rssMB‚ 2862.765625 disk.dm-0.in‚5.3125 disk.rbd2.in‚0$ memory_percent‚96.92511558532715' proc.memory.availableMB‚ 503.7890625 proc.cpu.threads‚17 disk.rbd1.out‚ 4.66259765625 network.sent‚ 2.4459087e+07 disk.sda.out‚ 446.3515625 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚446.3125! disk.rbd2.out‚270.13134765625 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.dm-1.in‚ 7.31640625 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚43567.02197265625 disk.sda1.out‚0) proc.memory.percent‚17.47293472290039 disk.rbd3.in‚0 disk.sda3.in‚ 7.31640625 disk.sda3.out‚446.3125' disk./.usagePercent‚73.739397908783 disk.sda.in‚ 7.3671875 cpu‚25.30484357397201 disk.rbd3.out‚0# disk.dm-0.out‚43567.02197265625 disk.drbd1777.in‚ 1.03515625 disk.sda2.in‚0oÚvÁ&‚(:… ŠîäÔ”«µx gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û~yuÂ&‚(:ê ŠîäÔŠíÂ| disk.dm-1.out‚ 446.765625 disk.rbd0.out‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd4.out‚0 disk.rbd2.out‚270.2412109375 disk.nvme0c0n1.in‚5.3125 network.sent‚ 2.4557017e+07 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚43567.83544921875 disk.dm-0.in‚5.3125) proc.memory.percent‚17.47293472290039% disk./.usageGB‚1098.7703170776367 disk.sda.in‚ 7.3671875 disk.dm-1.in‚ 7.31640625 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.in‚0$ memory_percent‚96.92447185516357( proc.memory.availableMB‚ 503.89453125 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0( disk.nvme0c0n1.out‚43567.83544921875 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 cpu‚25.30445482562735 network.recv‚ 3.331298e+06' disk./.usagePercent‚73.739397908783# disk.dm-0.out‚43567.83544921875 disk.rbd2.in‚0 disk.sda.out‚ 446.8046875 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 446.765625 disk.rbd1.out‚ 4.66259765625! proc.memory.rssMB‚ 2862.765625ðåK2Ã&- _wandbruntime‚9479  _runtime‚9479KV›~ÆÄ&À & lr-AdamW/pg1‚8.819720721313815e-05 & lr-AdamW/pg2‚8.819720721313815e-05 trainer/global_step‚9749 $ _timestamp‚1.788426008721298e+09 _runtime‚9479.471216269 _step‚800QÙ(¶ËÅ&Å & lr-AdamW/pg1‚8.819720721313815e-05 & lr-AdamW/pg2‚8.819720721313815e-05 trainer/global_step‚9749 $ _timestamp‚1.788426008721298e+09 _runtime‚9479.471216269 _step‚800 l†N”2Æ&- _wandbruntime‚9479  _runtime‚9479K"9î'Ç&èO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚366437653957632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18117557791744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚433668090386944 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚405104289304576 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386986977719296 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 253950347 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 180300147 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18117557791744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚405104535511040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18431903200768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 253976017 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 253950347 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 434250494 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 253976017 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚384502385132104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 180300149 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18117329588224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386986959716352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 180300147 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚433668387342848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚9749 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 434276166 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105490378 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18064954823620 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚405104535511040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105490376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415236211126272 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18064731174472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386986959716352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18432176216576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚366437670996992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0JÌÌÈ&íO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚433668387342848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚9749 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 434276166 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105490378 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18064954823620 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚405104535511040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105490376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415236211126272 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18064731174472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386986959716352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18432176216576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚366437670996992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚366437653957632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18117557791744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚433668090386944 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚405104289304576 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386986977719296 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 253950347 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reµ£c”Ýserved_bytes.all.allocated‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 180300147 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18117557791744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚405104535511040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18431903200768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 253976017 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 253950347 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 434250494 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 253976017 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚384502385132104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 180300149 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18117329588224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386986959716352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 180300147¡XbD\É&‚(:„ ™îäÔšª˜x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~£ëæzÊ&‚(:ï ™îäÔºùåz disk.rbd0.out‚0 disk.rbd1.out‚ 4.66259765625 disk.sda.out‚ 447.44921875 disk.sda3.in‚ 7.31640625 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625! proc.memory.rssMB‚ 2862.765625 network.recv‚ 3.336041e+06$ disk./.usageGB‚1098.770248413086 disk.rbd3.out‚0 disk.dm-1.out‚ 447.41015625( disk.nvme0c0n1.out‚43666.93798828125 disk.rbd1.in‚0# disk.dm-0.out‚43666.93798828125 disk.sda2.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0' proc.memory.availableMB‚ 503.2421875 cpu‚25.30401028079227) disk./.usagePercent‚73.73939330064738 disk.sda1.out‚0& disk.nvme0n1.out‚43666.93798828125 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.dm-1.in‚ 7.31640625$ memory_percent‚96.92845344543457 network.sent‚ 2.4563458e+07 disk.sda.in‚ 7.3671875 disk.sda3.out‚ 447.41015625 disk.dm-0.in‚5.3125 disk.rbd4.in‚0 disk.rbd4.out‚0) proc.memory.percent‚17.47293472290039 disk.rbd3.in‚0 disk.nvme0n1.in‚5.3125 disk.rbd2.in‚0 disk.rbd2.out‚271.8154296875_´ï 2Ë&- _wandbruntime‚9480  _runtime‚9480£vïyù&Ì&óM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 434293764 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387025297660416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚384540760678276 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18066563574072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚802 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 180317868 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 253975898 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 180317868 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚366473956415488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 254001594 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18119167282688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65826644 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 253975898 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 180317866 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 434319462 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 288553984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 434293764 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415277356882432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 % _timestamp‚1.7884260095931823e+09 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _runtime‚9480.343806029 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18119167282688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚602 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18066787223428 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 254001594 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 171327977 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171327375 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18119395499520 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18434037401600 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18433772787712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚366473973454848 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387025315663360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18119395499520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65827244 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚405144711162880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387025297660416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 434319462 ^þ&Í&øM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 180317866 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 434319462 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 288553984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 434293764 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415277356882432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 % _timestamp‚1.7884260095931823e+09 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _runtime‚9480.343806029 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18119167282688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚602 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18066787223428 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 254001594 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 171327977 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171327375 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18119395499520 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18434037401600 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18433772787712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚366473973454848 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387025315663360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18119395499520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65827244 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚405144711162880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387025297660416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 434319462 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 434293764 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387025297660416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚384540760678276 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18066563574072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚802 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 180317868 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 253975898 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 180317868 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚366473956415488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 254001594 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18119167282688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65826644 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 253975898¢Ã-D2Î&- _wandbruntime‚9480  _runtime‚9480€Ž C]Ï&× trainer/global_step‚9749 1loss/FBt2/assignment_loss‚0.16903516291876342 % _timestamp‚1.7884260095951893e+09 0loss/FRt2/assignment_loss‚0.9436356904251235 2loss/SRqqt2/assignment_loss‚0.7313214675865739 &loss/total_loss‚40.954078230995044 1loss/FBt4/assignment_loss‚0.07418381867215171 /loss/FRt2/detection_loss‚1.2135879143274255 .loss/FRt4/detection_loss‚2.427131412201905 /loss/FBt2/detection_loss‚0.5598609943169718 1loss/FBt1/assignment_loss‚0.46954624994077676 0loss/FBt3/assignment_loss‚1.9082553532595437 .loss/FRt1/detection_loss‚2.424333438679979 /loss/FRt1/assignment_loss‚3.732735830384332 1loss/SRqqt1/assignment_loss‚1.677353345354398 1loss/SRqqt4/detection_loss‚1.9636806554737545 epoch‚8 2loss/SRqqt3/assignment_loss‚1.2741805792227388 2loss/SRqqt4/assignment_loss‚1.1301188825496606 /loss/FRt3/detection_loss‚2.5858085248205396 0loss/SRqqt1/detection_loss‚2.360971727967262 1loss/SRqqt2/detection_loss‚0.8387846390935866 /loss/FBt3/detection_loss‚3.0908296704292297 /loss/FBt4/detection_loss‚1.9895201410566057 0loss/FRt4/assignment_loss‚2.2564622146742686 1loss/SRqqt3/detection_loss‚1.6008683888614177 _runtime‚9480.345029794 _step‚803 /loss/FRt3/assignment_loss‚2.768853407435947 /loss/FBt1/detection_loss‚2.7630187213420867|N6ÑbÐ&Ü 0loss/FRt2/assignment_loss‚0.9436356904251235 2loss/SRqqt2/assignment_loss‚0.7313214675865739 &loss/total_loss‚40.954078230995044 1loss/FBt4/assignment_loss‚0.07418381867215171 /loss/FRt2/detection_loss‚1.2135879143274255 .loss/FRt4/detection_loss‚2.427131412201905 /loss/FBt2/detection_loss‚0.5598609943169718 1loss/FBt1/assignment_loss‚0.46954624994077676 0loss/FBt3/assignment_loss‚1.9082553532595437 .loss/FRt1/detection_loss‚2.424333438679979 /loss/FRt1/assignment_loss‚3.732735830384332 1loss/SRqqt1/assignment_loss‚1.677353345354398 1loss/SRqqt4/detection_loss‚1.9636806554737545 epoch‚8 2loss/SRqqt3/assignment_loss‚1.2741805792227388 2loss/SRqqt4/assignment_loss‚1.1301188825496606 /loss/FRt3/detection_loss‚2.5858085248205396 0loss/SRqqt1/detection_loss‚2.360971727967262 1loss/SRqqt2/detection_loss‚0.8387846390935866 /loss/FBt3/detection_loss‚3.0908296704292297 /loss/FBt4/detection_loss‚1.9895201410566057 0loss/FRt4/assignment_loss‚2.2564622146742686 1loss/SRqqt3/detection_loss‚1.6008683888614177 _runtime‚9480.345029794 _step‚803 /loss/FRt3/assignment_loss‚2.768853407435947 /loss/FBt1/detection_loss‚2.7630187213420867 trainer/global_step‚9749 1loss/FBt2/assignment_loss‚0.16903516291876342 % _timestamp‚1.7884260095951893e+09£“ãOßÑ&‚(:… ¨îäÔÔÀ·x gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.784' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1568 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ŒofhÒ&‚(:Ý ¨îäÔ’ë€{# disk.nvme0c0n1.out‚ 43766.953125 disk.sda1.out‚0 disk.sda.out‚ 448.0546875 disk.rbd2.out‚271.9560546875) proc.memory.percent‚17.47293472290039 proc.cpu.threads‚17 network.sent‚ 2.4660191e+07 disk.drbd1777.in‚ 1.03515625 disk.rbd3.out‚0 disk.rbd1.out‚ 4.66259765625 disk.dm-1.out‚ 448.015625 disk.rbd0.in‚0 disk.rbd0.out‚0( disk./.usagePercent‚73.7393881804967 disk.rbd4.in‚0 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 448.015625 disk.sda2.out‚ 0.0390625! disk.nvme0n1.out‚ 43766.953125 disk.dm-1.in‚ 7.31640625 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.sda.in‚ 7.3671875 disk.nvme0n1.in‚5.3125$ memory_percent‚96.92580699920654( proc.memory.availableMB‚ 503.67578125 network.recv‚ 3.341822e+06 disk.dm-0.out‚ 43766.953125 cpu‚25.303608935121083 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 disk.sda2.in‚0% disk./.usageGB‚1098.7701721191406! proc.memory.rssMB‚ 2862.765625 disk.drbd1777.out‚0¬Óåø‹Ó&‚(:€ ·îäÔßø—x gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.5' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø,""Ô&‚(:Þ ·îäÔ휼z disk.rbd4.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 43866.0078125 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda.in‚ 7.3671875$ memory_percent‚96.92730903625488 network.recv‚ 3.348925e+06 disk.dm-1.in‚ 7.31640625 disk.dm-1.out‚ 448.28125 disk.rbd1.out‚ 4.66259765625 disk.nvme0n1.in‚5.3125" disk.nvme0n1.out‚ 43866.0078125' proc.memory.availableMB‚ 503.4296875) proc.memory.percent‚17.47293472290039 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03âàL°G515625 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.rbd1.in‚0! proc.memory.rssMB‚ 2862.765625 disk.rbd2.out‚271.9560546875 disk.rbd3.out‚0 disk.sda3.out‚ 448.28125) disk./.usagePercent‚73.73936616384876% disk./.usageGB‚1098.7698440551758 disk.dm-0.in‚5.3125 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 448.3203125 disk.sda3.in‚ 7.31640625 cpu‚25.303213028910864 network.sent‚ 2.4758294e+07 disk.drbd1777.out‚0$ disk.nvme0c0n1.out‚ 43866.0078125 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd0.out‚0â*6h2Õ&- _wandbruntime‚9524  _runtime‚9524ü¡^–ÇÖ&Á & lr-AdamW/pg2‚8.460697523209966e-05 trainer/global_step‚9799 % _timestamp‚1.7884260540510023e+09 _runtime‚9524.800986355 _step‚804 & lr-AdamW/pg1‚8.460697523209966e-05ÿ¥ðÌ×&Æ & lr-AdamW/pg1‚8.460697523209966e-05 & lr-AdamW/pg2‚8.460697523209966e-05 trainer/global_step‚9799 % _timestamp‚1.7884260540510023e+09 _runtime‚9524.800986355 _step‚804¤`Þz2Ø&- _wandbruntime‚9524  _runtime‚9524Aœï'Ù&éO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407113085644800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚388903856919552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚417294671998976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18525365517312 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚386409382284620 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18156588394828 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 436443018 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 172154253 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66146125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚435820338666496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 255231249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18209228725248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 181186097 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 181186097 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 436417346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚652 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 181186099 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚388903874922496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18525642727424 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 172153601 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66145475 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407113085644800 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 181186099 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435820037516288 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _step‚805 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 255256919 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚368252776850432 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚9799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 % _timestamp‚1.7884260540517392e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚652 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 181186099 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚388903874922496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18525642727424 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 172153601 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66145475 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407113085644800 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 181186099 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435820037516288 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _step‚805 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 255256919 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚368252776850432 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚9799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 % _timestamp‚1.7884260540517392e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚386409382284620 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18156588394828 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 436443018 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 172154253 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66146125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚435820338666496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 255231249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18209228725248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 181186097 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 181186097 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 436417346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚388942194863616 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 181203818 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 181203818 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18211066419712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435863369332736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚386447276453568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18158197145280 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18527503697408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚368289096347648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18158420794636 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66152248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚368289079308288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 181203816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 436486314 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚388942212866560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚386447517142284 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 _step‚806 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18527232792064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172170101 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 181203816 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚417335865635328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884260549198008e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 255256800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407153507503104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 436460616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 436486314 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18211294636544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 255282496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 255256800 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚388942212866560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚388942212866560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚388942194863616 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 181203818 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 181203818 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18211066419712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435863369332736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚386447276453568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18158197145280 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18527503697408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚368289096347648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18158420794636 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66152248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚368289079308288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 181203816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 436486314 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚388942212866560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚386447517142284 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 _step‚806 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18527232792064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172170101 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 181203816 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚417335865635328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884260549198008e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 255256800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407153507503104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 436460616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 436486314 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18211294636544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 255282496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824¦hÚL×2à&- _wandbruntime‚9525  _runtime‚9525ÑÔWá&Ñ .loss/FRt3/detection_loss‚2.590165506390964 .loss/FRt4/detection_loss‚2.326593069172313 1loss/SRqqt2/detection_loss‚0.8704372943858757 0loss/SRqqt4/detection_loss‚1.958650435321033 /loss/FRt1/assignment_loss‚2.861469441133997 2loss/SRqqt1/assignment_loss‚1.5484871694934554 2loss/SRqqt2/assignment_loss‚0.8520594255197934 /loss/FRt3/assignment_loss‚3.273279863245347 0loss/FRt4/assignment_loss‚3.1177254915237427 0loss/FBt4/assignment_loss‚0.3540786327794194 /loss/FRt1/detection_loss‚2.2950168355651526 1loss/SRqqt3/detection_loss‚1.6089466105082206 .loss/FBt1/detection_loss‚1.987270975112915 epoch‚8 1loss/FBt1/assignment_loss‚0.15975114219127134 0loss/FBt2/assignment_loss‚0.1294832309153227 2loss/SRqqt4/assignment_loss‚1.4502334296703339 .loss/FBt3/detection_loss‚2.613427847623825 % _timestamp‚1.7884260549215326e+09 0loss/FRt2/assignment_loss‚0.9440724763003263 /loss/FRt2/detection_loss‚1.6289244370704348 1loss/SRqqt1/detection_loss‚2.3194708824157715 _step‚807 1loss/SRqqt3/assignment_loss‚1.413517833620842 0loss/FBt2/detection_loss‚0.44472329916474834 %loss/total_loss‚40.10986244350693 trainer/global_step‚9799 _runtime‚ 9525.6713539 0loss/FBt3/assignment_loss‚0.8277213905239478 /loss/FBt4/detection_loss‚2.5343557238578795- &!\â&Ö 1loss/SRqqt3/assignment_loss‚1.413517833620842 0loss/FBt2/detection_loss‚0.44472329916474834 %loss/total_loss‚40.10986244350693 trainer/global_step‚9799 _runtime‚ 9525.6713539 0loss/FBt3/assignment_loss‚0.8277213905239478 /loss/FBt4/detection_loss‚2.5343557238578795 .loss/FRt3/detection_loss‚2.590165506390964 .loss/FRt4/detection_loss‚2.326593069172313 1loss/SRqqt2/detection_loss‚0.8704372943858757 0loss/SRqqt4/detection_loss‚1.958650435321033 /loss/FRt1/assignment_loss‚2.861469441133997 2loss/SRqqt1/assignment_loss‚1.5484871694934554 2loss/SRqqt2/assignment_loss‚0.8520594255197934 /loss/FRt3/assignment_loss‚3.273279863245347 0loss/FRt4/assignment_loss‚3.1177254915237427 0loss/FBt4/assignment_loss‚0.3540786327794194 /loss/FRt1/detection_loss‚2.2950168355651526 1loss/SRqqt3/detection_loss‚1.6089466105082206 .loss/FBt1/detection_loss‚1.987270975112915 epoch‚8 1loss/FBt1/assignment_loss‚0.15975114219127134 0loss/FBt2/assignment_loss‚0.1294832309153227 2loss/SRqqt4/assignment_loss‚1.4502334296703339 .loss/FBt3/detection_loss‚2.613427847623825 % _timestamp‚1.7884260549215326e+09 0loss/FRt2/assignment_loss‚0.9440724763003263 /loss/FRt2/detection_loss‚1.6289244370704348 1loss/SRqqt1/detection_loss‚2.3194708824157715 _step‚807§Xäã&‚(:… ÕîäÔìÐœy gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è]mxä&‚(:í ÕîäÔþ§øz# disk.dm-0.out‚43966.08544921875 disk.sda2.in‚0( proc.memory.availableMB‚ 502.49609375 disk.rbd4.out‚0 disk.sda.in‚ 7.3671875 disk.sda.out‚ 449.421875 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 449.3828125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0n1.in‚5.3125 network.sent‚ 2.4861952e+07$ disk./.usageGB‚1098.770435333252 disk.rbd1.in‚0! proc.memory.rssMB‚ 2862.765625 network.recv‚ 3.360342e+06 disk.dm-1.out‚ 449.3828125! disk.rbd2.out‚272.06591796875 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.dm-0.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd1.out‚ 4.81103515625) disk./.usagePercent‚73.73940584501656 disk.dm-1.in‚ 7.31640625 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚43966.08544921875) proc.memory.percent‚17.47293472290039 cpu‚25.30241636253041 disk.rbd4.in‚0 disk.rbd3.out‚0$ memory_percent‚96.93300724029541 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 7.31640625& disk.nvme0n1.out‚43966.08544921875¦æ¢Žå&‚(:ƒ äîäÔÏÛŸx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.623 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ý£­tæ&‚(:é äîäÔ—½ö{ network.sent‚ 2.4959357e+07 network.recv‚ 3.36606e+06 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 disk.sda.out‚ 449.8515625 disk.sda3.out‚449.8125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.sda2.out‚ 0.0390625 disk.rbd3.out‚0 disk.rbd1.out‚ 4.81103515625 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.dm-1.out‚449.8125$ memory_percent‚96.93500995635986( proc.memory.availableMB‚ 502.16796875) proc.memory.percent‚17.47293472290039% disk./.usageGB‚1098.7705154418945 disk.sda1.in‚ 0.05078125# disk.dm-0.out‚44065.33447265625! proc.memory.rssMB‚ 2862.765625 cpu‚25.302015835548055 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625& disk.nvme0n1.out‚44065.33447265625 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚ 7.31640625! disk.rbd2.out‚272.14404296875 disk.rbd0.out‚0 disk.rbd4.in‚0( disk.nvme0c0n1.out‚44065.33447265625 disk.sda.in‚ 7.3671875 disk.sda3.in‚ 7.31640625 disk.rbd1.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚73.73941122117478ÅÞ"ç&‚(:„ óîäÔ÷ê¢x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.503' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒrPPwè&‚(:ì óîäÔ¢®º{ network.recv‚ 3.370666e+06 disk.rbd3.in‚0 disk.sda.in‚ 7.3671875 disk.sda.out‚ 450.46484375 disk.rbd1.in‚0 disk.dm-1.in‚ 7.31640625) proc.memory.percent‚17.47293472290039 cpu‚25.301598395270332& disk.nvme0c0n1.out‚44164.443359375 disk.rbd1.out‚ 4.84326171875 proc.cpu.threads‚17 network.sent‚ 2.4965388e+07 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 450.42578125$ disk.nvme0n1.out‚44164.443359375$ memory_percent‚96.93360328674316% disk./.usageGB‚1098.7705841064453! disk.rbd2.out‚273.71826171875 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda2.in‚0 disk.dm-1.out‚ 450.42578125) disk./.usagePercent‚73.73941582931039 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125! disk.dm-0.out‚44164.443359375 disk.sda3.in‚ 7.31640625 disk.dm-0.in‚5.3125' proc.memory.availableMB‚ 502.3984375! proc.memory.rssMB‚ 2862.765625 disk.rbd4.in‚0•ÏGR2é&- _wandbruntime‚9570  _runtime‚9570\7çÇê&Á & lr-AdamW/pg2‚8.108271228848706e-05 trainer/global_step‚9849 % _timestamp‚1.7884260992873695e+09 _runtime‚9570.037263876 _step‚808 & lr-AdamW/pg1‚8.108271228848706e-05Žrš$Ìë&Æ & lr-AdamW/pg1‚8.108271228848706e-05 & lr-AdamW/pg2‚8.108271228848706e-05 trainer/global_step‚9849 % _timestamp‚1.7884260992873695e+09 _runtime‚9570.037263876 _step‚808¨ Vì2ì&- _wandbruntime‚9570  _runtime‚9570Š¦…"ï'í&éO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 438609870 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437971927808512 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚409122128191488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18301127862272 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18618770996736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚388316138748628 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 trainer/global_step‚9849 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 438584198 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 256537821 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419353180751872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 182072049 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66470752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 264627200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 256512151 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18247998316888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 % _timestamp‚1.7884260992881079e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409121881985024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18301356065792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419353156811776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚390820754122752 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66470122 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18301356065792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 256537821 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 _step‚809 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 256512151 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 182072047 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106525878 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437972216375808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚390820772125696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 288567296 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18301127862272 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18618770996736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚388316138748628 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 trainer/global_step‚9849 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 438584198 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 256537821 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419353180751872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 182072049 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66470752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 264627200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 256512151 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18247998316888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 % _timestamp‚1.7884260992881079e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409121881985024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 33µ7kéà7968128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18301356065792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419353156811776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚390820754122752 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66470122 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18301356065792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 256537821 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 _step‚809 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 256512151 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 182072047 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106525878 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437972216375808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚390820772125696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 288567296 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106536231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18302965556736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 173013382 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 _runtime‚9570.903953149 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18302965556736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66476466 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚409162303843328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18620927809536 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 _step‚810 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 256537702 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 438627468 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚390859092066816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚438014966635008 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182089766 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 438653166 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311622656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18620640126976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 $ _timestamp‚1.788426100153336e+09 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚370104202201088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚390859110069760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 438627468 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚9849 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18303193773568 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 256537702 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182089768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173012697 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18249830716488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66477149 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚388354032917576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18250054365844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚438015278257664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚409162303843328 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 182089766 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561.Eéý&ñ&÷M ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚370104202201088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚390859110069760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 438627468 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚9849 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18303193773568 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 256537702 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182089768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173012697 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18249830716488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66477149 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚388354032917576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18250054365844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚438015278257664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚409162303843328 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 182089766 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚390859110069760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106536231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18302965556736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 173013382 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 _runtime‚9570.903953149 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18302965556736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66476466 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚409162303843328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18620927809536 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 _step‚810 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 256537702 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 438627468 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚390859092066816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚438014966635008 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182089766 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 438653166 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311622656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18620640126976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 $ _timestamp‚1.788426100153336e+09 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2ª‚ †!2ò&- _wandbruntime‚9570  _runtime‚9570sãhWó&Ñ .loss/FRt4/detection_loss‚2.289487557117663 0loss/SRqqt1/detection_loss‚2.314473954744117 1loss/FBt3/assignment_loss‚0.32940666005015373 1loss/FBt4/assignment_loss‚0.25419344417750833 /loss/FRt2/detection_loss‚2.2371470564045013 .loss/FBt4/detection_loss‚3.114127290248871 epoch‚8 _step‚811 0loss/FRt2/assignment_loss‚1.8789047300815582 1loss/SRqqt1/assignment_loss‚1.709773343662883 1loss/FBt2/assignment_loss‚0.10564752389120613 .loss/FBt1/detection_loss‚1.755407614633441 2loss/SRqqt4/assignment_loss‚1.4082560855085435 1loss/SRqqt3/detection_loss‚1.7472115471027792 /loss/FBt2/detection_loss‚0.7028069955740648 -loss/FBt3/detection_loss‚1.67939130961895 %loss/total_loss‚41.09515767402364 0loss/FRt4/assignment_loss‚3.2029316425323486 2loss/SRqqt2/assignment_loss‚0.8934956842322687 2loss/SRqqt3/assignment_loss‚1.3560437289221834 trainer/global_step‚9849 _runtime‚9570.905096622 0loss/FRt3/assignment_loss‚2.7428519073522315 /loss/FRt1/detection_loss‚2.2913660045999746 .loss/FRt3/detection_loss‚2.275186394745449 0loss/SRqqt4/detection_loss‚2.042995257222134 % _timestamp‚1.7884261001552517e+09 /loss/FBt1/assignment_loss‚0.513983543604354 1loss/SRqqt2/detection_loss‚0.9079867916083649 /loss/FRt1/assignment_loss‚3.342081606388092´2µ\ô&Ö /loss/FRt1/assignment_loss‚3.342081606388092 .loss/FRt4/detection_loss‚2.289487557117663 0loss/SRqqt1/detection_loss‚2.314473954744117 1loss/FBt3/assignment_loss‚0.32940666005015373 1loss/FBt4/assignment_loss‚0.25419344417750833 /loss/FRt2/detection_loss‚2.2371470564045013 .loss/FBt4/detection_loss‚3.114127290248871 epoch‚8 _step‚811 0loss/FRt2/assignment_loss‚1.8789047300815582 1loss/SRqqt1/assignment_loss‚1.709773343662883 1loss/FBt2/assignment_loss‚0.10564752389120613 .loss/FBt1/detection_loss‚1.755407614633441 2loss/SRqqt4/assignment_loss‚1.4082560855085435 1loss/SRqqt3/detection_loss‚1.7472115471027792 /loss/FBt2/detection_loss‚0.7028069955740648 -loss/FBt3/detection_loss‚1.67939130961895 %loss/total_loss‚41.09515767402364 0loss/FRt4/assignment_loss‚3.2029316425323486 2loss/SRqqt2/assignment_loss‚0.8934956842322687 2loss/SRqqt3/assignment_loss‚1.3560437289221834 trainer/global_step‚9849 _runtime‚9570.905096622 0loss/FRt3/assignment_loss‚2.7428519073522315 /loss/FRt1/detection_loss‚2.2913660045999746 .loss/FRt3/detection_loss‚2.275186394745449 0loss/SRqqt4/detection_loss‚2.042995257222134 % _timestamp‚1.7884261001552517e+09 /loss/FBt1/assignment_loss‚0.513983543604354 1loss/SRqqt2/detection_loss‚0.9079867916083649«.{7­™õ&‚(:Ž ‚ïäÔÙû¤x gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.01' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.802000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0::’'rö&‚(:ç ‚ïäÔºÔÖ{ network.recv‚ 3.374853e+06 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 disk.dm-1.in‚ 7.31640625 disk.sda1.in‚ 0.05078125' proc.memory.availableMB‚ 502.3046875 network.sent‚ 2.4971742e+07) disk./.usagePercent‚73.73941582931039" disk.dm-0.out‚44263.7353515625 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda3.in‚ 7.31640625 disk.rbd3.out‚0! proc.memory.rssMB‚ 2862.765625) proc.memory.percent‚17.47293472290039 disk.nvme0c0n1.in‚5.3125' disk.nvme0c0n1.out‚44263.7353515625 disk.sda.in‚ 7.3671875 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0% disk.nvme0n1.out‚44263.7353515625! disk.rbd2.out‚273.79638671875 disk.sda1.out‚0 cpu‚25.30123062420378 disk.rbd1.out‚ 4.84326171875 disk.dm-0.in‚5.3125 disk.sda.out‚ 451.1953125 disk.dm-1.out‚ 451.15625 disk.rbd2.in‚0$ memory_percent‚96.93417549133301% disk./.usageGB‚1098.7705841064453 disk.sda3.out‚ 451.15625 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 proc.cpu.threads‚17]pöZ‹÷&‚(:€ ‘ïäÔ—‚šx gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.5' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô’:uø&‚(:ê ‘ïäÔÃØåz disk.sda.out‚ 451.55859375 network.sent‚ 2.5080729e+07 network.recv‚ 3.382341e+06$ memory_percent‚96.93272113800049 disk.dm-1.out‚ 451.51953125 disk.nvme0n1.in‚5.3125% disk.nvme0n1.out‚44362.7822265625 disk.rbd0.out‚0) proc.memory.percent‚17.47293472290039 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625" disk.dm-0.out‚44362.7822265625! disk.rbd2.out‚273.79638671875 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125' disk.nvme0c0n1.out‚44362.7822265625 disk.sda1.in‚ 0.05078125( proc.memory.availableMB‚ 502.54296875 disk.drbd1777.in‚ 1.03515625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 451.51953125 disk.sda1.out‚0 disk.dm-1.in‚ 7.31640625% disk./.usageGB‚1098.7706336975098 disk.rbd1.out‚4.90625 disk.sda2.in‚0! proc.memory.rssMB‚ 2862.765625 cpu‚25.300817918158444 disk.dm-0.in‚5.3125 disk.rbd4.out‚0 disk.sda.in‚ 7.3671875) disk./.usagePercent‚73.73941915740834 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0肾Þù&‚(:…  ïäÔÝ“©x gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž#Óú&‚(:ã  ïäÔزå{ disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.3671875# disk.dm-0.out‚44363.24755859375 disk.sda2.out‚ 0.0390625% disk./.usageGB‚1098.7689781188965 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda3.in‚ 7.31640625% proc.memory.avaQ™÷PPilableMB‚ 502.28125 network.sent‚ 2.5086362e+07) disk./.usagePercent‚73.73930805013849 disk.sda1.out‚0 disk.sda.out‚ 452.015625 disk.rbd1.out‚4.90625& disk.nvme0n1.out‚44363.24755859375 disk.rbd3.in‚0 cpu‚25.300422295809085 disk.dm-1.in‚ 7.31640625 disk.rbd2.out‚ 273.90234375( disk.nvme0c0n1.out‚44363.24755859375 disk.sda3.out‚ 451.9765625! proc.memory.rssMB‚ 2862.765625) proc.memory.percent‚17.47293472290039 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0 disk.sda2.in‚0$ memory_percent‚96.93431854248047 network.recv‚ 3.385632e+06 disk.dm-1.out‚ 451.9765625 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125FÞ€Ï2û&- _wandbruntime‚9615  _runtime‚9615ÉžÅOÇü&Á & lr-AdamW/pg1‚7.762515283341555e-05 & lr-AdamW/pg2‚7.762515283341555e-05 trainer/global_step‚9899 % _timestamp‚1.7884261445850391e+09 _runtime‚9615.334983182 _step‚812«y¡nÌý&Æ & lr-AdamW/pg2‚7.762515283341555e-05 trainer/global_step‚9899 % _timestamp‚1.7884261445850391e+09 _runtime‚9615.334983182 _step‚812 & lr-AdamW/pg1‚7.762515283341555e-05¬ÚG+2þ&- _wandbruntime‚9615  _runtime‚9615= ,@ï'ÿ&éO YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 trainer/global_step‚9899 _runtime‚9615.336004827 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18339855537244 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18339631888096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182957999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 182957997 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392737669328896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392737651325952 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18712601397760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390222654524128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚392737669328896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚411130924531712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 257818723 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚371883039675392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚813 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411130678325248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421411641624576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 257793053 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107043626 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚411130678325248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 1738401001)Dô'€'îO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18712326284800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚371883022636032 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 182957999 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 173839472 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18393026999296 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392737651325952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440123967909376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18339855537244 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18339631888096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182957999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 182957997 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392737669328896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392737651325952 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18712601397760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390222654524128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚392737669328896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚411130924531712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 257818723 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚371883039675392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚813 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411130678325248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421411641624576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 257793053 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107043626 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚411130678325248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 173840100 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 trainer/global_step‚9899 _runtime‚9615.336004827 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 440820018 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182975718 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18714194521088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚371919325093888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392775989270016 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107053981 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173856137 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9616.188350708 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 440820018 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107053983 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚614 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390261030070300 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 440794320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440167005841920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 257818604 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 % _timestamp‚1.7884261454379323e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182975716 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18394864693760 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18341464287696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0$ Ò«eƒ'øM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 % _timestamp‚1.7884261454379323e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182975716 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18394864693760 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18341464287696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_bÎSK™#atch_end/inactive_split_bytes.small_pool.peak‚ 309050880 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 440820018 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182975718 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18714194521088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚371919325093888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392775989270016 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107053981 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173856137 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9616.188350708 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 440820018 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107053983 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚614 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390261030070300 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 440794320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440167005841920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 257818604®þ›Á2„'- _wandbruntime‚9616  _runtime‚9616ýk,X…'Ò .loss/FBt2/detection_loss‚0.598931952154842 /loss/FBt4/detection_loss‚2.7927623748779298 .loss/FRt1/detection_loss‚2.404769315943122 1loss/SRqqt4/detection_loss‚1.8606082857871542 .loss/FRt1/assignment_loss‚3.22772578895092 _runtime‚9616.189471869 _step‚815 /loss/FRt2/assignment_loss‚1.303840394113578 1loss/FBt4/assignment_loss‚0.44152391944080593 .loss/FRt4/detection_loss‚2.496926924364134 0loss/SRqqt3/detection_loss‚1.598173852477755 trainer/global_step‚9899 0loss/FRt3/assignment_loss‚2.4054813553457675 2loss/SRqqt3/assignment_loss‚1.3332348915947867 /loss/FBt3/detection_loss‚2.1093344589074454 2loss/SRqqt4/assignment_loss‚1.7522768953016825 1loss/SRqqt2/detection_loss‚0.8382025099144532 /loss/FBt1/detection_loss‚1.9744246043264866 epoch‚8 % _timestamp‚1.7884261454396622e+09 0loss/FRt4/assignment_loss‚3.2853851379492345 0loss/FBt1/assignment_loss‚0.5022094018877397 0loss/FBt2/assignment_loss‚0.0825073893772351 0loss/FBt3/assignment_loss‚0.2803186151043822 /loss/FRt2/detection_loss‚1.5568182072510905 1loss/SRqqt1/detection_loss‚2.2769464270936117 %loss/total_loss‚39.57389698284222 2loss/SRqqt1/assignment_loss‚1.0128332059313025 1loss/SRqqt2/assignment_loss‚1.123565040442806 /loss/FRt3/detection_loss‚2.3150960343039553÷6-]†'× /loss/FRt3/detection_loss‚2.3150960343039553 .loss/FBt2/detection_loss‚0.598931952154842 /loss/FBt4/detection_loss‚2.7927623748779298 .loss/FRt1/detection_loss‚2.404769315943122 1loss/SRqqt4/detection_loss‚1.8606082857871542 .loss/FRt1/assignment_loss‚3.22772578895092 _runtime‚9616.189471869 _step‚815 /loss/FRt2/assignment_loss‚1.303840394113578 1loss/FBt4/assignment_loss‚0.44152391944080593 .loss/FRt4/detection_loss‚2.496926924364134 0loss/SRqqt3/detection_loss‚1.598173852477755 trainer/global_step‚9899 0loss/FRt3/assignment_loss‚2.4054813553457675 2loss/SRqqt3/assignment_loss‚1.3332348915947867 /loss/FBt3/detection_loss‚2.1093344589074454 2loss/SRqqt4/assignment_loss‚1.7522768953016825 1loss/SRqqt2/detection_loss‚0.8382025099144532 /loss/FBt1/detection_loss‚1.9744246043264866 epoch‚8 % _timestamp‚1.7884261454396622e+09 0loss/FRt4/assignment_loss‚3.2853851379492345 0loss/FBt1/assignment_loss‚0.5022094018877397 0loss/FBt2/assignment_loss‚0.0825073893772351 0loss/FBt3/assignment_loss‚0.2803186151043822 /loss/FRt2/detection_loss‚1.5568182072510905 1loss/SRqqt1/detection_loss‚2.2769464270936117 %loss/total_loss‚39.57389698284222 2loss/SRqqt1/assignment_loss‚1.0128332059313025 1loss/SRqqt2/assignment_loss‚1.123565040442806¯ñ‹|‡'‚(:… ¯ïäÔÒ™šx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.507' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h–¬Âsˆ'‚(:è ¯ïäÔ®áþz disk.sda1.out‚0 disk.sda.out‚ 452.55859375$ disk./.usageGB‚1098.770881652832 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.sda.in‚ 7.3671875 network.sent‚ 2.5092446e+07) disk./.usagePercent‚73.73943579789807 disk.rbd1.out‚4.90625 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚44462.38427734375 disk.dm-0.in‚5.3125 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚44462.38427734375 disk.rbd3.out‚0! proc.memory.rssMB‚ 2862.765625) proc.memory.percent‚17.47293472290039 proc.cpu.threads‚17 disk.sda3.out‚ 452.51953125# disk.dm-0.out‚44462.38427734375 disk.sda2.in‚0 disk.rbd2.out‚ 273.90234375( proc.memory.availableMB‚ 502.21484375 disk.sda3.in‚ 7.31640625 disk.rbd1.in‚0 disk.dm-1.out‚ 452.51953125 disk.rbd2.in‚0 disk.rbd0.out‚0$ memory_percent‚96.93472385406494 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚ 7.31640625 disk.rbd0.in‚0 cpu‚25.30003761462343 network.recv‚ 3.390273e+06 disk.rbd4.out‚02Ä' ‰'‚(:… ¾ïäÔÅóx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vŒMäuŠ'‚(:ê ¾ïäÔ¢œ´{ disk.rbd0.in‚0) proc.memory.percent‚17.47293472290039 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 7.31640625 network.sent‚ 2.5201318e+07 disk.dm-1.in‚ 7.31640625 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.3671875 disk.sda3.out‚ 453.00390625 disk.dm-0.in‚5.3125' disk.nvme0c0n1.out‚44562.0068359375 cpu‚25.299647585311803 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 453.00390625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0 disk.sda.out‚ 453.04296875 disk.nvme0n1.in‚5.3125 disk.rbd1.in‚0 disk.rbd1.out‚ 4.96923828125" disk.dm-0.out‚44562.0068359375# memory_percent‚96.9362735748291 network.recv‚ 3.397395e+06 disk.rbd2.out‚ 275.61328125 disk.rbd4.out‚0 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚44562.0068359375' proc.memory.availableMB‚ 501.9609375! proc.memory.rssMB‚ 2862.765625 proc.cpu.threads‚17) disk./.usagePercent‚73.73943579789807$ disk./.usageGB‚1098.770881652832ˆék‹'‚(:… ÍïäÔÿõ¶x gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.011' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\Œ /vŒ'‚(:ë ÍïäÔÿËêz disk.rbd1.out‚ 4.96923828125 disk.drbd1777.out‚0 disk.rbd4.out‚0' disk.nvme0c0n1.out‚44661.0302734375 network.recv‚ 3.402108e+06 disk.rbd2.in‚0 disk.sda.in‚ 7.3671875 cpu‚25.29923828750839 disk.rbd0.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0) proc.memory.percent‚17.47293472290039 disk.sda3.out‚ 453.62109375 disk.dm-1.out‚ 453.62109375 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚ 453.66015625$ memory_percent‚96.93772792816162( proc.memory.availableMB‚ 501.72265625 disk.sda3.in‚ 7.31640625 disk.rbd1.in‚0% disk.nvme0n1.out‚44661.0302734375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0! proc.memory.rssMB‚ 2862.765625) disk./.usagePercent‚73.73943579789807$ disk./.usageGB‚1098.770881652832 disk.dm-0.in‚5.3125" disk.dm-0.out‚44661.0302734375 disk.sda2.in‚0 disk.dm-1.in‚ 7.31640625 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 275.61328125 disk.rbd4.in‚0 proc.cpu.threads‚17 network.sent‚ 2.5207755e+07Œñ˜g2'- _wandbruntime‚9660  _runtime‚9660 ÿV­ÆŽ'À & lr-AdamW/pg1‚7.423501741709306e-05 & lr-AdamW/pg2‚7.423501741709306e-05 trainer/global_step‚9949 % _timestamp‚1.7884261898874407e+09 _runtime‚ 9660.63745762 _step‚816 V§Ë'Å & lr-AdamW/pg1‚7.423501741709306e-05 & lr-AdamW/pg2‚7.423501741709306e-05 trainer/global_step‚9949 % _timestamp‚1.7884261898874407e+09 _runtime‚ 9660.63745762 _step‚816°sÙk2'- _wandbruntime‚9660  _runtime‚9660h2Üï'‘'éO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 442943574 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚413139720871936 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚413139474665472 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174679633 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18484926136320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392129410988136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18484926136320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423470126437376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392129651676644 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174680272 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107561376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67118894 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373698145528832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 259073955 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183843949 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 183843947 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚9949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 442917902 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18485154339840 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18805819649536 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 259099625 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394654566532096 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 183843949 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚817 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 183843947 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 259099625 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800Ý1A_’'îO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 259073955 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183843949 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 183843947 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚9949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 442917902 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18485154339840 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18805819649536 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 259099625 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394654566532096 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 183843949 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚817 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 183843947 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 259099625 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 442943574 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚413139720871936 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚413139474665472 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174679633 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18484926136320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392129410988136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18484926136320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423470126437376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392129651676644 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174680272 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107561376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67118894 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373698145528832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299±×¥’Ù2“'- _wandbruntime‚9661  _runtime‚9661Á<ù&”'óM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 259099506 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚394692904476160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174696433 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚392167545845592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚394692886473216 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 183861666 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18433097858904 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 442986870 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚413179896523776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373734465026048 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18807959486464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 442986870 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18486763830784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67125344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 259125202 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 442961172 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚442318980520448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18807684386816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18433321508260 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 259099506 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18486992047616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183861668 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 183861666 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚442319279560192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚392167786534308 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚442319279560192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚392167786534308 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174696433 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚392167545845592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚394692886473216 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 183861666 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18433097858904 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 442986870 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚413179896523776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373734465026048 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18807959486464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 442986870 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18486763830784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67125344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 259125202 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 442961172 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚442318980520448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18807684386816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18433321508260 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 259099506 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18486992047616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183861668 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 183861666 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚9706.136209088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175520149 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415148271005696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415148271005696 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚396571463735296 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚396571445732352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 260354857 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚396571463735296 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 184729897 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67441023 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚444427973316096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108079128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 445084754 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175520783 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚375513268421632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 184729897 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 445110426 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚9999 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 260380527 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415148517212160 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18577053476864 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 184729899 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 260380527 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚415148517212160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108079126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67441655 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚444428266077696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425528635190272 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18899362065920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18523122679660 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18522899030512 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18899630887424 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18576825273344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚396571445732352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚375513285460992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚394036408140652 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 260354857 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18576825273344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884262353860395e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 184729899 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428T³I›ô'¤'îO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415148271005696 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚396571463735296 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚396571445732352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 260354857 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚396571463735296 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 184729897 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67441023 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚444427973316096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108079128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 445084754 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175520783 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚375513268421632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 184729897 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 445110426 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚9999 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 260380527 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415148517212160 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18577053476864 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 184729899 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 260380527 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚415148517212160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108079126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67441655 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚444428266077696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425528635190272 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18899362065920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18523122679660 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18522899030512 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18899630887424 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18576825273344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚396571445732352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚375513285460992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚394036408140652 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 260354857 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18576825273344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884262353860395e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 184729899 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚394036167452144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18577053476864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425528611250176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚9706.136209088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175520149 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415148271005696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168µ®½Üå2¥'- _wandbruntime‚9707  _runtime‚9707îUù&¦'óM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 260406104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚396609783676416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18578662967808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18901501165568 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 445153722 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 184747616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 260406104 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18524731430112 trainer/global_step‚9999 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 260380408 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚444471306052096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚415188446644224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 445128024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415188692864000 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚394074302309600 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175536728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚375549570879488 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 445153722 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425569804886528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚375549587918848 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚415188692864000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67447904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18901223968768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _step‚822 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 _runtime‚9707.005212533 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚444471004915200 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18578662967808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18901501165568 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 445153722 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 184747616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocajÆúÉžted‚ 260406104 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18524731430112 trainer/global_step‚9999 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 260380408 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚444471306052096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚415188446644224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 445128024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415188692864000 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚394074302309600 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175536728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚375549570879488 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 445153722 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425569804886528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚375549587918848 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚415188692864000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67447904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18901223968768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _step‚822 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 _runtime‚9707.005212533 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚444471004915200 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417157067345920 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚398488360938496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18668724410368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67767202 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚377328391314432 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 261635759 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395942923916152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108596878 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚395943164604660 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚10049 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417157313552384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚398488342935552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚9751.491730508 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚417157313552384 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 447277278 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚659 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 185615847 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚398488342935552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚427587120003072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18614532601720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18614756250868 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚417157067345920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608Kž„’¶'ïO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417157313552384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚398488342935552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚9751.491730508 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚417157313552384 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 447277278 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚659 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 185615847 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚398488342935552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚427587120003072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18614532601720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18614756250868 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚417157067345920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185615847 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚427587096062976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 261635759 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18668952613888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 % _timestamp‚1.7884262807415924e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚377328408353792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/active.smŒ¯õâ all_pool.allocated‚ 261661429 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 185615849 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚661 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18668724410368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚446579864844800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67766543 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚446580168092160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18993048089088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18992768781824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417157067345920 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚398488360938496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18668724410368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67767202 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚377328391314432 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 261635759 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395942923916152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108596878 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚395943164604660 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚10049¹Dn2·'- _wandbruntime‚9752  _runtime‚9752¸,ú&¸'ôM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚427628265759232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 _runtime‚9752.355292507 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185633566 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398526680879616 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176379908 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108607233 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417197489204224 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚446623199078912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚395981299462324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚377364710811648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417197242984448 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18616365001320 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚395981058773608 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18616588650676 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 trainer/global_step‚10049 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 % _timestamp‚1.7884262816049907e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427628289699328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 447294876 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67772677 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398526698882560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18994909379584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 261661310 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚398526698882560 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚398526680879616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 447320574 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185633566 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417197489204224 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 176380547 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 108607231 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 261687006 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 447320574 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 185633568 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561®Míêÿ&¹'ùM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 261661310 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚398526698882560 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚398526680879616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 447320574 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185633566 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417197489204224 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 176380547 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 108607231 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 261687006 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 447320574 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 185633568 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚427628265759232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 _runtime‚9752.355292507 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185633566 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398526680879616 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176379908 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108607233 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417197489204224 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚446623199078912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚395981299462324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚377364710811648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417197242984448 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18616365001320 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚395981058773608 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18616588650676 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 trainer/global_step‚10049 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 % _timestamp‚1.7884262816049907e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427628289699328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 447294876 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67772677 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398526698882560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18994909379584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561ºÆšZm2º'- _wandbruntime‚9752  _runtime‚9752XWT¤Z»'Ô 0loss/FRt3/assignment_loss‚2.8240044179062047 0loss/FBt4/assignment_loss‚0.5169984972869445 /loss/FRt2/detection_loss‚1.6674649562029278 .loss/FBt1/detection_loss‚2.493009070555369 epoch‚8 _step‚827 2loss/SRqqt2/assignment_loss‚0.8476935251746909 0loss/FBt2/assignment_loss‚0.1452117713923369 .loss/FRt3/detection_loss‚2.274856493497888 -loss/FRt4/detection_loss‚2.19026141176592 1loss/SRqqt3/detection_loss‚1.8184325292445065 .loss/FBt3/detection_loss‚2.405273951590061 2loss/SRqqt4/assignment_loss‚1.5292894738785765 0loss/FBt1/assignment_loss‚0.1989287988263868 0loss/FRt1/assignment_loss‚3.3876246350152153 1loss/SRqqt1/assignment_loss‚1.814229390025139 .loss/FRt1/detection_loss‚2.840247986997877 trainer/global_step‚10049 % _timestamp‚1.7884262816065836e+09 1loss/FBt3/assignment_loss‚0.02654513630841393 0loss/SRqqt2/detection_loss‚1.309025250338018 /loss/FBt4/detection_loss‚2.4309075559888567 &loss/total_loss‚42.260810691329326 2loss/SRqqt3/assignment_loss‚1.4010682713828588 /loss/FBt2/detection_loss‚0.6300396130865683 _runtime‚9752.356406336 0loss/FRt4/assignment_loss‚3.5373045802116394 1loss/SRqqt1/detection_loss‚2.3617319484551746 1loss/SRqqt4/detection_loss‚2.0213130959805024 0loss/FRt2/assignment_loss‚1.5893483302172493ˆVæZ_¼'Ù 2loss/SRqqt2/assignment_loss‚0.8476935251746909 0loss/FBt2/assignment_loss‚0.1452117713923369 .loss/FRt3/detection_loss‚2.274856493497888 -loss/FRt4/detection_loss‚2.19026141176592 1loss/SRqqt3/detection_loss‚1.8184325292445065 .loss/FBt3/detection_loss‚2.405273951590061 2loss/SRqqt4/assignment_loss‚1.5292894738785765 0loss/FBt1/assignment_loss‚0.1989287988263868 0loss/FRt1/assignment_loss‚3.3876246350152153 1loss/SRqqt1/assignment_loss‚1.814229390025139 .loss/FRt1/detection_loss‚2.840247986997877 trainer/global_step‚10049 % _timestamp‚1.7884262816065836e+09 1loss/FBt3/assignment_loss‚0.02654513630841393 0loss/SRqqt2/detection_loss‚1.309025250338018 /loss/FBt4/detection_loss‚2.4309075559888567 &loss/total_loss‚42.260810691329326 2loss/SRqqt3/assignment_loss‚1.4010682713828588 /loss/FBt2/detection_loss‚0.6300396130865683 _runtime‚9752.356406336 0loss/FRt4/assignment_loss‚3.5373045802116394 1loss/SRqqt1/detection_loss‚2.3617319484551746 1loss/SRqqt4/detection_loss‚2.0213130959805024 0loss/FRt2/assignment_loss‚1.5893483302172493 0loss/FRt3/assignment_loss‚2.8240044179062047 0loss/FBt4/assignment_loss‚0.5169984972869445 /loss/FRt2/detection_loss‚1.6674649562029278 .loss/FBt1/detection_loss‚2.493009070555369 epoch‚8 _step‚827»ô§½'‚(:„ ¶ðäÔ‡¼¥x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.783' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Kx÷r¾'‚(:ç ¶ðäÔš¿à{ network.recv‚ 3.435528e+06% disk.nvme0n1.out‚45164.2861328125 disk.sda3.out‚ 457.03125 disk.nvme0n1.in‚5.3125 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.rbd3.out‚0! proc.memory.rssMB‚ 2862.765625 disk.dm-1.out‚ 457.03125) disk./.usagePercent‚73.73743637905565 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚277.6650390625% disk./.usageGB‚1098.7410888671875 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0) proc.memory.percent‚17.47293472290039 disk.dm-1.in‚ 7.31640625 network.sent‚ 2.5453124e+07 disk.rbd1.out‚ 4.96923828125 disk.sda.in‚ 7.3671875 disk.sda3.in‚ 7.31640625( proc.memory.availableMB‚ 336.31640625 disk.rbd4.in‚0' disk.nvme0c0n1.out‚45164.2861328125" disk.dm-0.out‚45164.2861328125 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda.out‚ 457.0703125 cpu‚25.29656064987523 proc.cpu.threads‚17 disk.nvme0c0n1.in‚5.3125$ memory_percent‚97.94728755950928îC囿'‚(: ÅðäÔ鈱x gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/“VwÀ'‚(:ì ÅðäÔŒü{ disk.rbd4.out‚0 cpu‚25.296187407791233 network.sent‚ 2.5562239e+07 disk.sda1.out‚0 disk.rbd4.in‚0 network.recv‚ 3.443323e+06 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚45263.39990234375 disk.sda.in‚ 7.3671875& disk.nvme0n1.out‚45263.39990234375 disk.sda2.in‚0 disk.rbd0.in‚0$ memory_percent‚97.94812202453613) disk./.usagePercent‚73.73743637905565 disk.sda.out‚ 457.90625 disk.drbd1777.in‚ 1.03515625' proc.memory.availableMB‚ 336.1796875 proc.cpu.threads‚17 disk.sda3.out‚ 457.8671875 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125) proc.memory.percent‚17.47293472290039 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚5.3125# disk.dm-0.out‚45263.39990234375 disk.dm-1.in‚ 7.31640625 disk.dm-1.out‚ 457.8671875 disk.rbd2.out‚277.6845703125 disk.sda3.in‚ 7.31640625 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2862.765625% disk./.usageGB‚1098.7410888671875 disk.rbd1.out‚ 4.96923828125 disk.nvme0n1.in‚5.3125cuûÄ›Á'‚(: ÔðäÔ‹ì’x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.467' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.293400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥¯«©yÂ'‚(:î ÔðäÔåÌÊ{ network.recv‚ 3.447057e+06( disk./.usagePercent‚73.7374397071536 disk.rbd1.out‚ 4.96923828125 disk.rbd4.out‚0) proc.memory.percent‚17.47293472290039 network.sent‚ 2.5567974e+07 disk.sda2.in‚0! proc.memory.rssMB‚ 2862.765625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 458.41796875& disk.nvme0n1.out‚45363.37646484375 disk.rbd2.out‚277.7470703125 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 458.37890625 disk.rbd1.in‚0 disk.dm-1.in‚ 7.31640625 disk.dm-1.out‚ 458.37890625 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚45363.37646484375 cpu‚25.29578299423551 disk.sda.in‚ 7.3671875 disk.rbd2.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚1098.741138458252 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125# disk.dm-0.out‚45363.37646484375 disk.rbd0.out‚0 disk.rbd4.in‚0$ memory_percent‚97.94812202453613' proc.memory.availableMB‚ 336.1796875 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.sda1.out‚0»¬ì2Ã'- _wandbruntime‚9796  _runtime‚9796wB¸ ÇÄ'Á & lr-AdamW/pg1‚6.447614925623116e-05 & lr-AdamW/pg2‚6.447614925623116e-05 trainer/global_step‚10099 $ _timestamp‚1.788426326015838e+09 _runtime‚9796.765783621 _step‚828KoqÌÅ'Æ _runtime‚9796.765783621 _step‚828 & lr-AdamW/pg1‚6.447614925623116e-05 & lr-AdamW/pg2‚6.447614925623116e-05 trainer/global_step‚10099 $ _timestamp‚1.788426326015838e+09¼'5­32Æ'- _wandbruntime‚9796  _runtime‚9796ÿÅ(ÏlÇ'êO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 186501799 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 281404416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177207041 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109114628 _runtime‚9796.766757917 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419165863686144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419165863686144 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19086369473024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚397849680380160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 449444130 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 262942331 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚448731974288896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400405240138752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚397849921068668 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 186501797 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚429645580875776 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 262916661 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177207721 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18760623547392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18760851750912 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18706389822076 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 186501797 % _timestamp‚1.7884263260166194e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 449418458 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 262916661 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 trainer/global_step‚10099 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19086088068608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109114626 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 _step‚829 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177207041 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109114628 _runtime‚9796.766757917 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419165863686144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 305344512 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419165863686144 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19086369473024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚397849680380160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 449444130 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 262942331 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚448731974288896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400405240138752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚397849921068668 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 186501797 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚429645580875776 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 262916661 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177207721 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18760623547392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18760851750912 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18706389822076 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 186501797 % _timestamp‚1.7884263260166194e+09½’@š2É'- _wandbruntime‚9797  _runtime‚9797­0 ú&Ê'ôM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚419206285544448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 _runtime‚9797.632163497 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 177224601 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 262942212 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚400443596085760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18762689458688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18707998572528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18762461241856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 449487426 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 186519516 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19088229063168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68098962 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 262967908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚397887815237616 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 186519518 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177223943 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚397888055926332 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19087962352128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚429686774512128 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 449487426 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 262942212 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884263268815765e+09 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109124981 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 449487426 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 186519516 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19088229063168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68098962 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 262967908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚397887815237616 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 186519518 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177223943 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚397888055926332 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inacti›'ªÇve_split.all.current‚658 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19087962352128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚429686774512128 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 449487426 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 262942212 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884263268815765e+09 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109124981 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 177224601 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 262942212 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚400443596085760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18762689458688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18707998572528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18762461241856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187387747 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19179949748224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18852522684416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚421174906232832 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19179683023872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 % _timestamp‚1.7884263715791354e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚402322155344896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚402322137341952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚833 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18852522684416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178051807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68418790 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109632376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚9842.329237369 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421174660026368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 264223233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚431704065688576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 264197563 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 451610982 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚399756436844168 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 187387749 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18852750887936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402322155344896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚450884039376896 trainer/global_step‚10149 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 264223233 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚402322137341952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187387749 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚450883748712448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚421174906232832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18798023393284 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18797799744136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421174660026368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178051166ï–-•ËÚ'ïO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚399756436844168 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 187387749 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18852750887936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402322155344896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚450884039376896 trainer/global_step‚10149 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 264223233 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚402322137341952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187387749 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚450883748712448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚421174906232832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18798023393284 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18797799744136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421174660026368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178051166 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380958637100032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109632378 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187387747 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19179949748224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18852522684416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚421174906232832 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19179683023872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 % _timestamp‚1.7884263715791354e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚402322155344896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚402322137341952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚833 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freedŽBôm* ‚18852522684416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178051807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68418790 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109632376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _runtime‚9842.329237369 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421174660026368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 264223233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚431704065688576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 264197563 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 451610982 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624Á+–D2Û'- _wandbruntime‚9843  _runtime‚9843‰p§ù&Ü'óM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 187405468 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 451654278 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68425443 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚431745235384832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚402360493288960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18799855793092 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 264248810 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178068174 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19181549694464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109642733 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚402360475286016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚10149 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18854588595712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402360475286016 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18854588595712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚431745259324928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚399794571701624 $ _timestamp‚1.788426372449042e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚380994956597248 _runtime‚9843.199689184 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚834 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚450926785079296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19181820599808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18854360378880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109642731 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚380994939557888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178068820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19181820599808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18854360378880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109642731 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚380994939557888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178068820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 451654278 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68425443 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚431745235384832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚402360493288960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18799855793092 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 264248810 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178068174 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19181549694464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109642733 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚402360475286016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚10149 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18854588595712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402360475286016 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18854588595712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚431745259324928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚399794571701624 $ _timestamp‚1.788426372449042e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚380994956597248 _runtime‚9843.199689184 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚834 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚450926785079296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800·¬›2Þ'- _wandbruntime‚9843  _runtime‚9843¬Z)MYß'Ó 2loss/SRqqt2/assignment_loss‚0.6682471508301527 0loss/SRqqt1/detection_loss‚2.143577814930015 0loss/SRqqt3/detection_loss‚1.745715231001377 /loss/FBt4/detection_loss‚2.6697893738746643 /loss/FRt3/assignment_loss‚3.076419069323429 2loss/SRqqt4/assignment_loss‚1.5002553719420766 /loss/FRt2/detection_loss‚1.9816774389706553 % _timestamp‚1.7884263724510045e+09 /loss/FRt1/assignment_loss‚2.849392810533213 /loss/FRt3/detection_loss‚2.4852522091810094 0loss/FBt2/assignment_loss‚0.1729632305920539 /loss/FRt4/detection_loss‚2.1608767766000154 1loss/SRqqt2/detection_loss‚0.9046541013184094 trainer/global_step‚10149 2loss/SRqqt3/assignment_loss‚1.6226319875568151 %loss/total_loss‚40.42547812471743 _step‚835 _runtime‚9843.200823501 /loss/FBt4/assignment_loss‚0.321487646739115 /loss/FRt1/detection_loss‚2.3820592733316643 1loss/SRqqt4/detection_loss‚1.8993497795836871 /loss/FBt1/detection_loss‚2.6952894926071167 .loss/FBt3/detection_loss‚2.025321227312088 epoch‚8 1loss/FBt1/assignment_loss‚0.11223990644118333 0loss/FBt3/assignment_loss‚0.3920470055192709 .loss/FBt2/detection_loss‚0.529459601739444 0loss/FRt2/assignment_loss‚1.6041945770382882 /loss/FRt4/assignment_loss‚3.124591246895168 2loss/SRqqt1/assignment_loss‚1.3579858008565173„@¨e^à'Ø /loss/FRt3/detection_loss‚2.4852522091810094 0loss/FBt2/assignment_loss‚0.1729632305920539 /loss/FRt4/detection_loss‚2.1608767766000154 1loss/SRqqt2/detection_loss‚0.9046541013184094 trainer/global_step‚10149 2loss/SRqqt3/assignment_loss‚1.6226319875568151 %loss/total_loss‚40.42547812471743 _step‚835 _runtime‚9843.200823501 /loss/FBt4/assignment_loss‚0.321487646739115 /loss/FRt1/detection_loss‚2.3820592733316643 1loss/SRqqt4/detection_loss‚1.8993497795836871 /loss/FBt1/detection_loss‚2.6952894926071167 .loss/FBt3/detection_loss‚2.025321227312088 epoch‚8 1loss/FBt1/assignment_loss‚0.11223990644118333 0loss/FBt3/assignment_loss‚0.3920470055192709 .loss/FBt2/detection_loss‚0.529459601739444 0loss/FRt2/assignment_loss‚1.6041945770382882 /loss/FRt4/assignment_loss‚3.124591246895168 2loss/SRqqt1/assignment_loss‚1.3579858008565173 2loss/SRqqt2/assignment_loss‚0.6682471508301527 0loss/SRqqt1/detection_loss‚2.143577814930015 0loss/SRqqt3/detection_loss‚1.745715231001377 /loss/FBt4/detection_loss‚2.6697893738746643 /loss/FRt3/assignment_loss‚3.076419069323429 2loss/SRqqt4/assignment_loss‚1.5002553719420766 /loss/FRt2/detection_loss‚1.9816774389706553 % _timestamp‚1.7884263724510045e+09 /loss/FRt1/assignment_loss‚2.849392810533213È ­›á'‚(: ñäÔÀî¹x gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.979' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.195800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'GM@xâ'‚(:í ñäÔ²—ëz$ disk./.usageGB‚1098.741153717041 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.dm-0.in‚5.3125 proc.cpu.threads‚17% disk.nvme0n1.out‚45661.1533203125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚ 7.31640625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda3.out‚ 460.55859375 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.96923828125" disk.dm-0.out‚45661.1533203125 network.sent‚ 2.5694617e+07 disk.nvme0c0n1.in‚5.3125 disk.sda.in‚ 7.3671875 disk.sda2.in‚0 disk.dm-1.out‚ 460.55859375 disk.rbd0.out‚0' disk.nvme0c0n1.out‚45661.1533203125 disk.nvme0n1.in‚5.3125! proc.memory.rssMB‚ 2862.765625 disk.sda.out‚ 460.59765625! disk.rbd2.out‚279.31005859375) disk./.usagePercent‚73.73744073118374 disk.rbd3.out‚0 disk.drbd1777.out‚0( proc.memory.availableMB‚ 335.72265625) proc.memory.percent‚17.47293472290039 cpu‚25.29429450290217 disk.rbd4.in‚0 disk.rbd4.out‚0$ memory_percent‚97.95091152191162 disk.sda3.in‚ 7.31640625 network.recv‚ 3.46692e+06Áœûâã'‚(:„ ŸñäÔÒ±¼x gpu.0.gpu‚21.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚88.973' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7946 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç }ä'‚(:ò ŸñäÔâ¼þz disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 461.18359375 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd4.out‚0$ disk./.usageGB‚1098.741153717041 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 cpu‚25.293923417411186 disk.rbd3.out‚0 disk.sda.in‚ 7.3671875 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚45661.41845703125 disk.dm-0.in‚5.3125# disk.dm-0.out‚45661.41845703125 disk.sda.out‚ 461.22265625 disk.rbd1.in‚0$ memory_percent‚97.94938564300537 disk.dm-1.out‚ 461.18359375 disk.rbd2.in‚0( proc.memory.availableMB‚ 335.97265625! proc.memory.rssMB‚ 2862.765625) proc.memory.percent‚17.47293472290039 proc.cpu.threads‚17 disk.rbd4.in‚0 network.sent‚ 2.5803335e+07 network.recv‚ 3.474259e+06 disk.rbd3.in‚0! disk.rbd2.out‚279.31787109375 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚45661.41845703125 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.96923828125 disk.drbd1777.in‚ 1.03515625 disk.dm-1.in‚ 7.31640625 disk.rbd0.out‚0) disk./.usagePercent‚73.73744073118374¨B¬±å'‚(:… ®ñäÔÈÛ‚x gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚102.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b·QÍ{æ'‚(:ð ®ñäÔѽÂ{ disk.rbd3.out‚0! proc.memory.rssMB‚ 2862.765625 cpu‚25.293551514589453# disk.dm-0.out‚45761.03564453125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd4.out‚0$ memory_percent‚97.95088768005371& disk.nvme0n1.out‚45761.03564453125 disk.sda.out‚ 461.66015625) proc.memory.percent‚17.47293472290039$ disk./.usageGB‚1098.741153717041 disk.sda3.out‚ 461.62109375 disk.nvme0n1.in‚5.3125 disk.dm-1.out‚ 461.62109375 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚45761.03564453125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚ 7.31640625! disk.rbd2.out‚279.45556640625 disk.rbd0.in‚0 disk.rbd3.in‚0 network.sent‚ 2.580935e+07 network.recv‚ 3.478622e+06 disk.sda3.in‚ 7.31640625 disk.rbd1.in‚0 disk.rbd1.out‚ 4.96923828125 disk.dm-0.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.sda.in‚ 7.3671875' proc.memory.availableMB‚ 335.7265625) disk./.usagePercent‚73.73744073118374 disk.rbd0.out‚0M/¼ù2ç'- _wandbruntime‚9888  _runtime‚9888!˵Èè' & lr-AdamW/pg1‚5.831992843721984e-05 & lr-AdamW/pg2‚5.831992843721984e-05 trainer/global_step‚10199 % _timestamp‚1.7884264175261133e+09 _runtime‚9888.276053429 _step‚836³ÀÛÝÍé'Ç & lr-AdamW/pg1‚5.831992843721984e-05 & lr-AdamW/pg2‚5.831992843721984e-05 trainer/global_step‚10199 % _timestamp‚1.7884264175261133e+09 _runtime‚9888.276053429 _step‚836ÄÏÛð‡2ê'- _wandbruntime‚9888  _runtime‚9888Vöz## ë'éO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423183702573056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 453752162 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423183456366592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚404239052548096 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 188273697 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚382773777032192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚404239034545152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18944421821440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18944421821440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 265478465 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 188273699 _step‚837 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18889656964492{¥(Çô'ì'îO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 265504135 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401663433996684 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401663193308176 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18944650024960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚665 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚663 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚433762574441472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19273081113088 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423183456366592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚404239052548096 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 188273697 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚382773777032192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚404239034545152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18944421821440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18944421821440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 265478465 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 188273699 _step‚837 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18889656964492 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423183702573056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 453752162 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 453821130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 110160483 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 188291416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚453078964153856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 188291418 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 188291418 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423223632005120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚382810079490048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 265504016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚838 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18891489364300 trainer/global_step‚10199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68750688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚404277372489216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19275220016128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚401701568854348 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423223878224896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚ 9889.15199422 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚404277372489216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178911169 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18946259515904 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚433803744137728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚433803720197632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110160481 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 188291416 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178911807 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚453078665114112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19274944916480óé®òï'øM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚404277372489216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19275220016128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚401701568854348 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423223878224896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 _runtime‚ 9889.15199422 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚404277372489216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178911169 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18946259515904 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚433803744137728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚433803720197632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110160481 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 188291416 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 17891Î|Á 1807 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚453078665114112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19274944916480 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 453821130 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18946259515904 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 265529712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 453795432 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 453821130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 110160483 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 188291416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚453078964153856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 188291418 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 188291418 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423223632005120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚382810079490048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 265504016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚838 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18891489364300 trainer/global_step‚10199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68750688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243Æ9 i2ð'- _wandbruntime‚9889  _runtime‚9889ìmbZñ'Ô 0loss/FRt1/assignment_loss‚3.3439681204882534 0loss/FRt3/assignment_loss‚2.7256237971477018 2loss/SRqqt2/assignment_loss‚0.9867333470272922 2loss/FBt4/assignment_loss‚0.016915294545469806 .loss/FRt3/detection_loss‚2.585141761180682 .loss/FBt3/detection_loss‚2.026899553835392 trainer/global_step‚10199 $ _timestamp‚1.788426418403419e+09 0loss/FRt4/assignment_loss‚3.1636730054529703 1loss/FBt2/assignment_loss‚0.07744447776286362 0loss/FBt3/assignment_loss‚0.2943973111687228 1loss/SRqqt2/detection_loss‚0.8663940807023356 0loss/FRt2/assignment_loss‚1.5799749005924573 2loss/SRqqt1/assignment_loss‚1.7196569825437935 /loss/FRt1/detection_loss‚2.1113744819706137 .loss/FRt4/detection_loss‚2.285323610714931 0loss/FBt1/assignment_loss‚0.1601489662341671 /loss/FBt1/detection_loss‚2.2037148150530728 %loss/total_loss‚39.56519295620916 1loss/SRqqt3/assignment_loss‚1.362063991563285 1loss/SRqqt4/assignment_loss‚1.456379344952958 0loss/SRqqt1/detection_loss‚2.103700156577609 0loss/SRqqt4/detection_loss‚1.880361449353549 /loss/FBt4/detection_loss‚2.4389495477080345 epoch‚8 _step‚839 .loss/FRt2/detection_loss‚1.488952632045204 1loss/SRqqt3/detection_loss‚2.0823529935350606 /loss/FBt2/detection_loss‚0.6050483340527429 _runtime‚9889.153262299ïšJH_ò'Ù /loss/FBt4/detection_loss‚2.4389495477080345 epoch‚8 _step‚839 .loss/FRt2/detection_loss‚1.488952632045204 1loss/SRqqt3/detection_loss‚2.0823529935350606 /loss/FBt2/detection_loss‚0.6050483340527429 _runtime‚9889.153262299 0loss/FRt1/assignment_loss‚3.3439681204882534 0loss/FRt3/assignment_loss‚2.7256237971477018 2loss/SRqqt2/assignment_loss‚0.9867333470272922 2loss/FBt4/assignment_loss‚0.016915294545469806 .loss/FRt3/detection_loss‚2.585141761180682 .loss/FBt3/detection_loss‚2.026899553835392 trainer/global_step‚10199 $ _timestamp‚1.788426418403419e+09 0loss/FRt4/assignment_loss‚3.1636730054529703 1loss/FBt2/assignment_loss‚0.07744447776286362 0loss/FBt3/assignment_loss‚0.2943973111687228 1loss/SRqqt2/detection_loss‚0.8663940807023356 0loss/FRt2/assignment_loss‚1.5799749005924573 2loss/SRqqt1/assignment_loss‚1.7196569825437935 /loss/FRt1/detection_loss‚2.1113744819706137 .loss/FRt4/detection_loss‚2.285323610714931 0loss/FBt1/assignment_loss‚0.1601489662341671 /loss/FBt1/detection_loss‚2.2037148150530728 %loss/total_loss‚39.56519295620916 1loss/SRqqt3/assignment_loss‚1.362063991563285 1loss/SRqqt4/assignment_loss‚1.456379344952958 0loss/SRqqt1/detection_loss‚2.103700156577609 0loss/SRqqt4/detection_loss‚1.880361449353549ǵõq[Žó'‚(:ƒ ½ñäԾ˿x gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.77' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.954 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£¾Rtô'‚(:é ½ñäÔ¹ áz network.recv‚ 3.483356e+06$ disk./.usageGB‚1098.741153717041 disk.dm-0.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0) disk./.usagePercent‚73.73744073118374 disk.rbd3.in‚0 disk.rbd1.in‚0# disk.dm-0.out‚45860.14501953125( disk.nvme0c0n1.out‚45860.14501953125 cpu‚25.293187444864508 disk.sda1.out‚0 disk.nvme0n1.in‚5.3125 disk.rbd2.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda.out‚ 462.1953125! disk.rbd2.out‚279.45556640625 disk.rbd0.in‚0 disk.rbd4.in‚0( proc.memory.availableMB‚ 335.65234375) proc.memory.percent‚17.47293472290039 disk.sda.in‚ 7.3671875 disk.sda3.in‚ 7.31640625 disk.rbd1.out‚ 4.96923828125 disk.sda2.out‚ 0.0390625" memory_percent‚97.951340675354! proc.memory.rssMB‚ 2862.765625 disk.drbd1777.out‚0 network.sent‚ 2.5815658e+07 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 462.15625 disk.sda2.in‚0& disk.nvme0n1.out‚45860.14501953125 disk.dm-1.in‚ 7.31640625 disk.dm-1.out‚ 462.15625 disk.nvme0c0n1.in‚5.3125©»¥õ'‚(:… ÌñäÔƳÅx gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.457' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U9öoö'‚(:ä ÌñäÔ•óÕz disk.dm-0.in‚5.3125 disk.sda1.out‚0 disk.sda.in‚ 7.3671875 disk.sda3.in‚ 7.31640625! proc.memory.rssMB‚ 2862.765625 cpu‚25.292795840793058 disk.rbd3.out‚0 disk.sda2.out‚ 0.0390625 network.sent‚ 2.5923735e+07 network.recv‚ 3.489226e+06$ disk./.usageGB‚1098.741153717041 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125$ memory_percent‚97.94976711273193 disk.rbd3.in‚0 disk.nvme0n1.in‚5.3125 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚ 462.7421875 disk.dm-1.out‚ 462.703125) disk./.usagePercent‚73.73744073118374 disk.rbd1.in‚0 disk.dm-0.out‚45959.30859375 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd4.out‚0( proc.memory.availableMB‚ 335.91015625 disk.rbd1.out‚ 4.96923828125 disk.rbd0.out‚0 disk.sda3.out‚ 462.703125) proc.memory.percent‚17.47293472290039 proc.cpu.threads‚17 disk.dm-1.in‚ 7.31640625! disk.rbd2.out‚279.52587890625 disk.drbd1777.out‚0% disk.nvme0c0n1.out‚45959.30859375# disk.nvme0n1.out‚45959.30859375²²*›÷'‚(: ÛñäÔúÛ‚x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.974' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.594800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„kìQvø'‚(:ë ÛñäÔ¡µÙz disk.rbd1.in‚0 disk.rbd4.in‚0! proc.memory.rssMB‚ 2862.765625 proc.cpu.threads‚17 disk.dm-1.in‚ 7.31640625 disk.rbd1.out‚ 4.96923828125 disk.rbd0.out‚0 disk.sda.in‚ 7.3671875 disk.dm-1.out‚ 463.23046875 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 463.23046875% proc.memory.availableMB‚ 335.90625 disk.rbd3.out‚0 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda3.in‚ 7.31640625$ memory_percent‚97.94979095458984" disk.dm-0.out‚45959.5556640625 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.in‚0) proc.memory.percent‚17.47293472290039( disk./.usagePercent‚73.7373260398084$ disk./.usageGB‚1098.739444732666 disk.sda2.in‚0 disk.sda1.out‚0 network.sent‚ 2.5929795e+07% disk.nvme0n1.out‚45959.5556640625 disk.rbd4.out‚0 disk.sda.out‚ 463.26953125 cpu‚25.292429940071596 network.recv‚ 3.493682e+06 disk.dm-0.in‚5.3125! disk.rbd2.out‚279.52978515625' disk.nvme0c0n1.out‚45959.5556640625Ÿn„[2ù'- _wandbruntime‚9933  _runtime‚9933¼åbÈú' & lr-AdamW/pg1‚5.534867180812035e-05 & lr-AdamW/pg2‚5.534867180812035e-05 trainer/global_step‚10249 % _timestamp‚1.7884264630357296e+09 _runtime‚9933.785685827 _step‚84003ÖiÍû'Ç & lr-AdamW/pg1‚5.534867180812035e-05 & lr-AdamW/pg2‚5.534867180812035e-05 trainer/global_step‚10249 % _timestamp‚1.7884264630357296e+09 _runtime‚9933.785685827 _step‚840È÷/„2ü'- _wandbruntime‚9933  _runtime‚9933µ,{áð'ý'êO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚9933.786619727 _step‚841 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚406155931748352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425192252706816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384588899924992 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18981066886552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69069924 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455187574955008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384588882885632 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 266759367 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚435821059254272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18981290535700 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 266785037 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 179737802 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110667876 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19366539640832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 266785037 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189159649 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 179737164 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 455944686 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406155931748352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19366814753792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚435821035314176 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 % _timestamp‚1.7884264630364964e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406155949751296 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 trainer/global_step‚10249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425192252706816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 266759367 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19036320958464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚406155949751296 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 189159647 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110667878 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚455187874008064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19036549161984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69069288 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚403569949772184 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚425192498913280 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚425192498913280 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 455919014 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0`§a%ž#þ'ïO dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19366814753792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚435821035314176 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 % _timestamp‚1.7884264630364964e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406155949751296 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 trainer/global_step‚10249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425192252706816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 266759367 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19036320958464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚406155949751296 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 189159647 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110667878 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚455187874008064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19036549161984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69069288 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚403569949772184 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚425192498913280 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚425192498913280 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 455919014 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚9933.786619727 _step‚841 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚406155931748352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425192252706816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384588899924992 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18981066886552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69069924 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455187574955008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384588882885632 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 266759367 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚435821059254272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18981290535700 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 266785037 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 179737802 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110667876 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19366539640832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 266785037 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189159649 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 179737164 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 SDDeviceStatsMonitor.on_tr€ƒçWain_batch_start/active_bytes.large_pool.peak‚ 6252512256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 455944686 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406155931748352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520ÉG³–±2ÿ'- _wandbruntime‚9934  _runtime‚9934ð}/`ú&€(ôM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 455987982 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚406194287695360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189177366 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 266784918 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚9934.629602779 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚425232674565120 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 179754307 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19038386869760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189177366 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 266810614 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚435862205010432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19368406475264 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚403608325318356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 455962284 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384625185343488 trainer/global_step‚10249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110678231 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18983122935508 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19038158652928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406194269692416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚435862228950528 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 179753655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69076074 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 266784918 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19038386869760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚455230908428288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19038158652928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 455987982 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚842 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19368679477760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 266810614 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69075424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384625202382848 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18982899286152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚9934.629602779 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚425232674565120 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 179754307 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19038386869760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189177366 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 266810614 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚435862205010432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19368406475264 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚403608325318356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 455962284 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384625185343488 trainer/global_step‚10249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110678231 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18983122935508 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19038158652928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406194269692416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚435862228950528 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 179753655 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69076074 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 266784918 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19038386869760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚455230908428288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19038158652928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 455987982 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚842 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19368679477760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 266810614 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69075424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384625202382848 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18982899286152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 268065939 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 288567296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚e?"i18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427201295253504 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 180577527 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69391899 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437879520126976 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 190045597 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427201049047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 190045599 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 264627200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _step‚845 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚653 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 190045597 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 458085866 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚408072828951552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427201049047040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚408072828951552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19128220095488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 458111538 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19128220095488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 190045599 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 _runtime‚9979.239242795 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚408072846954496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19460093098496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0F±õ'(ïO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405476706236192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437879544067072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚405476946924700 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 268040269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457339901792768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427201295253504 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 180577527 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69391899 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437879520126976 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 190045597 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427201049047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 190045599 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 264627200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _step‚845 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚655 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚653 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 190045597 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 458085866 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚408072828951552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427201049047040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚408072828951552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19128220095488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 458111538 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19128220095488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 190045599 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 _runtime‚9979.239242795 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚408072846954496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19460093098496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 268065939 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111185626 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 268040269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚408072846954496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 268065939 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 288567296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 458154834 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69397726 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚405515081782364 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405514841093648 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19130286006784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 458129136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 190063318 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437920689823232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190063318 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408111184898560 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 458154834 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180593707 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚457382942998016 _step‚846 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386440308236288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19074532857360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19130057789952 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 268065820 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427241470905344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 268091516 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19462229234688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19130286006784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386440325275648 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 trainer/global_step‚10299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427241470905344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 180594339 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 190063316 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408111166895616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19130057789952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457382650249728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896X²ˆ¼“(ùM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19130286006784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 458129136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 190063318 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437920689823232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190063318 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408111184898560 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 458154834 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180593707 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚457382942998016 _step‚846 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386440308236288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19074532857360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19130057789952 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 268065820 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427241470905344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 268091516 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19462229234688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19130286006784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386440325275648 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 trainer/global_step‚10299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427241470905344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.°×-,Ccurrent‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 180594339 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 190063316 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408111166895616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19130057789952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457382650249728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19461960426496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 458154834 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69397726 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚405515081782364 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405514841093648 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824Î ÷ùÍ2”(- _wandbruntime‚9980  _runtime‚9980%å7_•(Ù 2loss/SRqqt2/assignment_loss‚0.8389367894119313 2loss/SRqqt4/assignment_loss‚1.4693534306057219 .loss/FBt1/detection_loss‚4.909969588120778 /loss/FRt2/assignment_loss‚1.724952383683278 0loss/FBt1/assignment_loss‚0.3874823385849595 .loss/FRt1/detection_loss‚2.829642590028899 1loss/SRqqt2/detection_loss‚0.9605501614991696 /loss/SRqqt4/detection_loss‚1.78431680123761 trainer/global_step‚10299 /loss/FRt2/detection_loss‚1.8229138560306568 /loss/FRt4/detection_loss‚2.3176625319202957 0loss/FBt2/detection_loss‚0.49799184872591423 epoch‚8 % _timestamp‚1.7884265093644114e+09 2loss/FBt3/assignment_loss‚0.018976850109174846 2loss/FBt4/assignment_loss‚0.061351904936600477 0loss/SRqqt1/detection_loss‚2.287306779309323 2loss/FBt2/assignment_loss‚0.048673394784919596 0loss/SRqqt3/detection_loss‚1.966604172859503 /loss/FBt3/detection_loss‚2.3979724407196046 _step‚847 0loss/FRt1/assignment_loss‚2.9775366221155437 2loss/SRqqt1/assignment_loss‚1.9153938775784092 .loss/FBt4/detection_loss‚2.068868100643158 2loss/SRqqt3/assignment_loss‚1.1428888273101463 /loss/FRt3/detection_loss‚2.4016996920108795 _runtime‚9980.114258671 &loss/total_loss‚43.012887392502954 /loss/FRt3/assignment_loss‚2.868231485949622 0loss/FRt4/assignment_loss‚3.3136109243268552Ìön d–(Þ /loss/SRqqt4/detection_loss‚1.78431680123761 trainer/global_step‚10299 /loss/FRt2/detection_loss‚1.8229138560306568 /loss/FRt4/detection_loss‚2.3176625319202957 0loss/FBt2/detection_loss‚0.49799184872591423 epoch‚8 % _timestamp‚1.7884265093644114e+09 2loss/FBt3/assignment_loss‚0.018976850109174846 2loss/FBt4/assignment_loss‚0.061351904936600477 0loss/SRqqt1/detection_loss‚2.287306779309323 2loss/FBt2/assignment_loss‚0.048673394784919596 0loss/SRqqt3/detection_loss‚1.966604172859503 /loss/FBt3/detection_loss‚2.3979724407196046 _step‚847 0loss/FRt1/assignment_loss‚2.9775366221155437 2loss/SRqqt1/assignment_loss‚1.9153938775784092 .loss/FBt4/detection_loss‚2.068868100643158 2loss/SRqqt3/assignment_loss‚1.1428888273101463 /loss/FRt3/detection_loss‚2.4016996920108795 _runtime‚9980.114258671 &loss/total_loss‚43.012887392502954 /loss/FRt3/assignment_loss‚2.868231485949622 0loss/FRt4/assignment_loss‚3.3136109243268552 2loss/SRqqt2/assignment_loss‚0.8389367894119313 2loss/SRqqt4/assignment_loss‚1.4693534306057219 .loss/FBt1/detection_loss‚4.909969588120778 /loss/FRt2/assignment_loss‚1.724952383683278 0loss/FBt1/assignment_loss‚0.3874823385849595 .loss/FRt1/detection_loss‚2.829642590028899 1loss/SRqqt2/detection_loss‚0.9605501614991696ÏËÛªá—(‚(:… —òäÔ ±x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.936' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·ÝŠv˜(‚(:ë —òäÔäÄðz disk.rbd3.out‚0 disk.rbd0.in‚0! disk.rbd2.out‚281.33056640625( disk.nvme0c0n1.out‚46257.71533203125 disk.rbd1.out‚ 4.96923828125# disk.dm-0.out‚46257.71533203125) disk./.usagePercent‚73.73733218398921 disk.rbd4.out‚0 disk.sda3.in‚ 7.31640625 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 cpu‚25.290951800972675 network.recv‚ 3.513023e+06 disk.sda2.in‚0 disk.dm-1.in‚ 7.31640625 disk.dm-1.out‚ 465.2578125 disk.rbd2.in‚0 network.sent‚ 2.60565e+07 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda.out‚ 465.296875& disk.nvme0n1.out‚46257.71533203125% disk./.usageGB‚1098.7395362854004 disk.rbd3.in‚0 disk.sda.in‚ 7.3671875 disk.sda3.out‚ 465.2578125# memory_percent‚97.9520320892334' proc.memory.availableMB‚ 335.5390625! proc.memory.rssMB‚ 2862.765625 disk.nvme0c0n1.in‚5.3125 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.drbd1777.out‚0) proc.memory.percent‚17.47293472290039 proc.cpu.threads‚17 disk.sda1.in‚ 0.05078125þ*| ™(‚(:… ¦òäÔø¸¤x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚107.171' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EŽÿ}š(‚(:ò ¦òäÔ͉ßz( disk.nvme0c0n1.out‚46356.97412109375 disk.sda3.out‚ 465.75390625 disk.nvme0n1.in‚5.3125$ memory_percent‚97.95241355895996' proc.memory.availableMB‚ 335.4765625% disk./.usageGB‚1098.7395362854004 disk.rbd0.in‚0 disk.sda3.in‚ 7.31640625 disk.rbd1.out‚ 4.96923828125# disk.dm-0.out‚46356.97412109375 proc.cpu.threads‚17! disk.rbd2.out‚283.02978515625 disk.sda1.out‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 465.75390625 disk.rbd4.in‚0 disk.sda.out‚ 465.79296875& disk.nvme0n1.out‚46356.97412109375 cpu‚25.290567314206488 disk.sda2.out‚ 0.0390625 disk.rbd3.in‚0 disk.dm-0.in‚5.3125 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.rbd1.in‚0 network.recv‚ 3.519924e+06 disk.sda2.in‚0 disk.sda1.in‚ 0.05078125! proc.memory.rssMB‚ 2862.765625) proc.memory.percent‚17.47293472290039 network.sent‚ 2.6165099e+07 disk.dm-1.in‚ 7.31640625 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.in‚ 7.3671875) disk./.usagePercent‚73.73733218398921 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0'J¹Ž›(‚(:ƒ µòäÔü×x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.26' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.652 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P2(3{œ(‚(:ð µòäÔŽÁ©z& disk.nvme0n1.out‚46456.04443359375 disk.rbd2.in‚0 disk.sda.out‚ 466.31640625 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.sda2.in‚0! disk.rbd2.out‚283.05712890625( proc.memory.availableMB‚ 335.47265625# disk./.usageGB‚1098.73921585083! proc.memory.rssMB‚ 2862.765625 proc.cpu.threads‚17 disk.rbd4.in‚0 disk.sda3.in‚ 7.31640625 disk.dm-1.in‚ 7.31640625 disk.rbd0.in‚0$ memory_percent‚97.95243740081787 disk.dm-1.out‚ 466.27734375 disk.rbd4.out‚0 disk.sda.in‚ 7.3671875# disk.dm-0.out‚46456.04443359375 disk.rbd0.out‚0 network.sent‚ 2.6171267e+07 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda2.out‚ 0.0390625 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.sda3.out‚ 466.27734375 disk.rbd1.out‚ 4.96923828125) proc.memory.percent‚17.47293472290039 cpu‚25.29021033874017 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 network.recv‚ 3.524184e+06) disk./.usagePercent‚73.73731067935634 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚46456.04443359375 disk.rbd1.in‚0ù)w'4(/ _wandbruntime‚10024  _runtime‚10024°ìËž(Å ' lr-AdamW/pg1‚4.9622962008832905e-05 ' lr-AdamW/pg2‚4.9622962008832905e-05 trainer/global_step‚10349 % _timestamp‚1.7884265538058064e+09 _runtime‚10024.555751319 _step‚848°…ð0П(Ê % _timestamp‚1.7884265538058064e+09 _runtime‚10024.555751319 _step‚848 ' lr-AdamW/pg1‚4.9622962008832905e-05 ' lr-AdamW/pg2‚4.9622962008832905e-05 trainer/global_step‚10349Ðt™ª4 (/ _wandbruntime‚10024  _runtime‚10024Æ8¥ï'¡(éO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 460278390 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429209845387264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19553851755520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚459491880635392 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 283501568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 $ _timestamp‚1.788426553806554e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429210091593728 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚388219145710592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚439938028879872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19553568253952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 190931549 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚409989744157696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19220119232512 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 460252718 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚409989726154752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69715948 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 269346841 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚407383703388708 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19164557678116 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19164334028968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚407383462700200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 181419324 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429209845387264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429209845387264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19553851755520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚459491880635392 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 283501568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 $ _timestamp‚1.788426553806554e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429210091593728 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚388219145710592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚439938028879872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19553568253952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 190931549 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚409989744157696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19220119232512 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 460252718 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚409989726154752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69715948 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 269346841 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚407383703388708 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19164557678116 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19164334028968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚407383462700200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 181419324 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429209845387264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 460278390 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚439979174636032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚410028064098816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚407421597557656 _runtime‚10025.420171143 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚429250267245568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 190949266 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19166390077924 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚407421838246372 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181435801 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 190949268 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚388255448168448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 269372418 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19555710321152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚388255431129088 $ _timestamp‚1.788426554669573e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190949268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69722070 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 trainer/global_step‚10349 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19222185143808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 269372418 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 181436439 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19221956926976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 460321686 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚410028064098816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69722706 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚410028082101760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚459534908897280 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19166166428568 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111713731 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 190949266 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 460295988 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111713733 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚459534609857536 _step‚850-<ÿ&¥(ùM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 190949266 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 460295988 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111713733 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚459534609857536 _step‚850 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚429250021025792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚439979174636032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚410028064098816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚407421597557656 _runtime‚10025.420171143 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚429250267245568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 190949266 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19166390077924 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚407421838246372 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181435801 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 190949268 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚388255448168448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 269372418 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19555710321152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚388255431129088 $ _timestamp‚1.788426554669573e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190949268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69722070 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 trainer/global_step‚10349 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19222185143808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 269372418 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 181436439 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19221956926976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 460321686 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚410028064098816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69722706 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚410028082101760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚459534908897280 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19166166428568 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111713731 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561ÒÕ Æ“4¦(/ _wandbruntime‚10025  _runtime‚10025Ñer\§(Ö 0loss/SRqqt4/detection_loss‚1.808590739654998 .loss/FBt1/detection_loss‚5.505814880132675 /loss/FBt2/detection_loss‚0.5811193858530244 /loss/FBt3/detection_loss‚2.8282724142074587 2loss/SRqqt4/assignment_loss‚1.3935322621061157 /loss/FRt3/detection_loss‚2.5371999740600586 /loss/FRt4/detection_loss‚2.5926141412485215 _step‚851 0loss/FRt1/assignment_loss‚3.4964081575704173 0loss/FRt2/assignment_loss‚1.3617551650689996 2loss/SRqqt1/assignment_loss‚1.8963728174567223 1loss/SRqqt3/assignment_loss‚1.280829459352761 /loss/FBt3/assignment_loss‚0.533981953188777 &loss/total_loss‚45.818852589415975 % _timestamp‚1.7884265546713815e+09 0loss/FRt4/assignment_loss‚3.0956564318566095 2loss/SRqqt2/assignment_loss‚0.9411321847287358 /loss/FRt1/detection_loss‚2.7196741887303286 0loss/SRqqt1/detection_loss‚2.052358311155568 /loss/FBt4/detection_loss‚2.6887867550055184 1loss/FBt1/assignment_loss‚0.47464916855096817 /loss/SRqqt3/detection_loss‚1.89343611379059 _runtime‚10025.421214939 /loss/FRt3/assignment_loss‚3.185076517528958 0loss/SRqqt2/detection_loss‚1.003045553738465 1loss/FBt2/assignment_loss‚0.07199319243244491 epoch‚8 trainer/global_step‚10349 1loss/FBt4/assignment_loss‚0.18254093348999353 /loss/FRt2/detection_loss‚1.6940118885072677ÝÌa¨(Û trainer/global_step‚10349 1loss/FBt4/assignment_loss‚0.18254093348999353 /loss/FRt2/detection_loss‚1.6940118885072677 0loss/SRqqt4/detection_loss‚1.808590739654998 .loss/FBt1/detection_loss‚5.505814880132675 /loss/FBt2/detection_loss‚0.5811193858530244 /loss/FBt3/detection_loss‚2.8282724142074587 2loss/SRqqt4/assignment_loss‚1.3935322621061157 /loss/FRt3/detection_loss‚2.5371999740600586 /loss/FRt4/detection_loss‚2.5926141412485215 _step‚851 0loss/FRt1/assignment_loss‚3.4964081575704173 0loss/FRt2/assignment_loss‚1.3617551650689996 2loss/SRqqt1/assignment_loss‚1.8963728174567223 1loss/SRqqt3/assignment_loss‚1.280829459352761 /loss/FBt3/assignment_loss‚0.533981953188777 &loss/total_loss‚45.818852589415975 % _timestamp‚1.7884265546713815e+09 0loss/FRt4/assignment_loss‚3.0956564318566095 2loss/SRqqt2/assignment_loss‚0.9411321847287358 /loss/FRt1/detection_loss‚2.7196741887303286 0loss/SRqqt1/detection_loss‚2.052358311155568 /loss/FBt4/detection_loss‚2.6887867550055184 1loss/FBt1/assignment_loss‚0.47464916855096817 /loss/SRqqt3/detection_loss‚1.89343611379059 _runtime‚10025.421214939 /loss/FRt3/assignment_loss‚3.185076517528958 0loss/SRqqt2/detection_loss‚1.003045553738465 1loss/FBt2/assignment_loss‚0.07199319243244491 epoch‚8Ó;©U›©(‚(: ÄòäÔ£«x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.467' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.293400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vª(xª(‚(:í ÄòäÔªâØ{% disk.nvme0n1.out‚46457.1708984375 disk.dm-1.out‚ 466.83203125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 466.83203125! proc.memory.rssMB‚ 2862.765625 disk.sda.in‚ 7.3671875 disk.nvme0n1.in‚5.3125$ memory_percent‚97.94869422912598 cpu‚25.28986994952365 network.sent‚ 2.6175811e+07 network.recv‚ 3.527737e+06 disk.dm-1.in‚ 7.31640625 disk.rbd0.out‚0 disk.rbd4.in‚0' disk.nvme0c0n1.out‚46457.1708984375 proc.cpu.threads‚17) disk./.usagePercent‚73.73730760726592 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0$ disk./.usageGB‚1098.739170074463 disk.sda.out‚ 466.87109375 disk.rbd1.out‚ 4.96923828125 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.dm-0.in‚5.3125' proc.memory.availableMB‚ 336.0859375! disk.rbd2.out‚284.71337890625" disk.dm-0.out‚46457.1708984375) proc.memory.percent‚17.47293472290039û5±X›«(‚(: ÓòäÔ‰§’x gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.969' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.593799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{^[s¬(‚(:è ÓòäÔÅÁºz network.sent‚ 2.6284746e+07 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 disk.dm-1.out‚ 467.28125 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125! proc.memory.rssMB‚ 2862.765625% disk.nvme0n1.out‚46556.1904296875 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0( proc.memory.availableMB‚ 336.32421875) disk./.usagePercent‚73.73730760726592 disk.dm-0.in‚5.3125" disk.dm-0.out‚46556.1904296875 disk.rbd2.in‚0! disk.rbd2.out‚284.73681640625 disk.rbd3.out‚0$ memory_percent‚97.94723987579346 disk.sda2.out‚ 0.0390625 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0$ disk./.usageGB‚1098.739170074463 disk.sda3.out‚ 467.28125 disk.rbd1.out‚ 4.96923828125 disk.rbd0.in‚0 disk.sda3.in‚ 7.31640625 disk.rbd1.in‚0 disk.dm-1.in‚ 7.31640625 disk.rbd4.out‚0 disk.sda1.out‚0 cpu‚25.289494487878784 network.recv‚ 3.535086e+06 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚46556.1904296875) proc.memory.percent‚17.47293472290039 proc.cpu.threads‚17 disk.sda.in‚ 7.3671875 disk.sda.out‚ 467.3203125va­(‚(:… âòäԺIJx gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.027' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@¹t®(‚(:é âòäÔ»æ¼z cpu‚25.289138406740676 disk.sda3.in‚ 7.31640625 disk.rbd1.out‚ 4.96923828125 disk.nvme0n1.in‚5.3125! disk.rbd2.out‚286.39697265625 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0' disk.nvme0c0n1.out‚46655.3115234375 network.sent‚ 2.6290631e+07 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625$ disk./.usageGB‚1098.739170074463 disk.sda2.in‚0 disk.dm-1.in‚ 7.31640625 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125( proc.memory.availableMB‚ 336.32421875) proc.memory.percent‚17.47293472290039 proc.cpu.threads‚17 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 467.7734375 disk.sda.in‚ 7.3671875 disk.sda.out‚467.8125$ memory_percent‚97.94723987579346) disk./.usagePercent‚73.73730760726592 disk.sda3.out‚ 467.7734375" disk.dm-0.out‚46655.3115234375 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 network.recv‚ 3.539155e+06% disk.nvme0n1.out‚46655.3115234375! proc.memory.rssMB‚ 2862.765625 disk.rbd0.out‚0 disk.rbd4.out‚0ynʸ4¯(/ _wandbruntime‚10069  _runtime‚10069#ÑHŒÉ°(à & lr-AdamW/pg1‚4.686970206798508e-05 & lr-AdamW/pg2‚4.686970206798508e-05 trainer/global_step‚10399 % _timestamp‚1.7884265991603873e+09 _runtime‚10069.910281774 _step‚852ÍÅ4α(È & lr-AdamW/pg1‚4.686970206798508e-05 & lr-AdamW/pg2‚4.686970206798508e-05 trainer/global_step‚10399 % _timestamp‚1.7884265991603873e+09 _runtime‚10069.910281774 _step‚852ÔÝ[4²(/ _wandbruntime‚10069  _runtime‚1006991³â–³(ëO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431218887933952 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 191817497 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 270602073 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112221128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409290459852716 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19255967600176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 270627743 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚390034251564032 _step‚853 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 191817499 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚431218887933952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411906641360896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70041618 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431218641727488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚411906623357952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70040988 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 270627743 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19312246573056 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 270602073Úù:Öö'´(ðO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411906641360896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70041618 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431218641727488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚411906623357952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70040988 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 270627743 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19312246573056 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 270602073 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431218887933952 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 191817497 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 270602073 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112221128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409290459852716 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19255967600176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 270627743 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚390034251564032 _step‚853 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 191817499 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚431218887933952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148Õõj·È4µ(/ _wandbruntime‚10070  _runtime‚10070o—E û&¶(õM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431258817366016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 270627624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19258023649132 _runtime‚10070.780712198 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182278613 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 _step‚854 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 191835216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70047760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚461686791412736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚409328354021664 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411944979304960 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19313856064000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 182279243 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚411944961302016 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚442037683388928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 191835218 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 462488538 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112231481 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70047132 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚442037659448832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409328594710380 % _timestamp‚1.7884266000303164e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 270653320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19649108023808 trainer/global_step‚10399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19648828729856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚411944961302016 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 270653320 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461686488178688Ao5]·(úM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚411944961302016 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 270653320 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461686488178688 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431258817366016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 270627624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19258023649132 _runtime‚10070.780712198 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182278613 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 _step‚854 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 191835216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70047760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚461686791412736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚409328354021664 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411944979304960 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19313856064000 PBDeviceStatsMonitor.on_train_batch_end/inactive_spliAÃ'p£ t.all.allocated‚ 182279243 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚411944961302016 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚442037683388928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 191835218 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 462488538 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112231481 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70047132 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚442037659448832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409328594710380 % _timestamp‚1.7884266000303164e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 270653320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19649108023808 trainer/global_step‚10399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19648828729856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073ÖuØÒ{4¸(/ _wandbruntime‚10070  _runtime‚10070܇Å[¹(Õ 2loss/SRqqt1/assignment_loss‚1.3619451065148627 1loss/FBt3/assignment_loss‚0.23575270688161254 /loss/FRt1/detection_loss‚2.6913581502598687 0loss/FRt1/assignment_loss‚3.0230079470454037 0loss/FRt4/assignment_loss‚2.9455130219459535 1loss/SRqqt4/assignment_loss‚1.387786390963528 trainer/global_step‚10399 .loss/FRt4/detection_loss‚2.160333674578562 /loss/FBt1/detection_loss‚2.8816334530711174 /loss/FRt2/detection_loss‚1.5497281389667632 /loss/FBt2/detection_loss‚0.6807616716371728 .loss/FBt4/detection_loss‚2.098757180902693 $ _timestamp‚1.788426600031905e+09 _runtime‚10070.78173794 .loss/FRt3/assignment_loss‚3.02657286713763 1loss/SRqqt2/detection_loss‚0.8206748214239875 0loss/SRqqt3/detection_loss‚1.755226102968057 _step‚855 1loss/FBt4/assignment_loss‚0.14753492686819905 /loss/FRt3/detection_loss‚2.5264601605694468 1loss/SRqqt4/detection_loss‚1.9017319020297792 epoch‚8 /loss/FRt2/assignment_loss‚0.856108252038347 2loss/SRqqt2/assignment_loss‚0.8340051613566983 2loss/SRqqt3/assignment_loss‚1.3793206598687295 0loss/FBt2/assignment_loss‚0.1058275936305209 &loss/total_loss‚40.256551549204325 1loss/FBt1/assignment_loss‚0.17820106367446442 0loss/SRqqt1/detection_loss‚2.623120869909014 /loss/FBt3/detection_loss‚3.0851897249619165È4`º(Ú 0loss/FRt1/assignment_loss‚3.0230079470454037 0loss/FRt4/assignment_loss‚2.9455130219459535 1loss/SRqqt4/assignment_loss‚1.387786390963528 trainer/global_step‚10399 .loss/FRt4/detection_loss‚2.160333674578562 /loss/FBt1/detection_loss‚2.8816334530711174 /loss/FRt2/detection_loss‚1.5497281389667632 /loss/FBt2/detection_loss‚0.6807616716371728 .loss/FBt4/detection_loss‚2.098757180902693 $ _timestamp‚1.788426600031905e+09 _runtime‚10070.78173794 .loss/FRt3/assignment_loss‚3.02657286713763 1loss/SRqqt2/detection_loss‚0.8206748214239875 0loss/SRqqt3/detection_loss‚1.755226102968057 _step‚855 1loss/FBt4/assignment_loss‚0.14753492686819905 /loss/FRt3/detection_loss‚2.5264601605694468 1loss/SRqqt4/detection_loss‚1.9017319020297792 epoch‚8 /loss/FRt2/assignment_loss‚0.856108252038347 2loss/SRqqt2/assignment_loss‚0.8340051613566983 2loss/SRqqt3/assignment_loss‚1.3793206598687295 0loss/FBt2/assignment_loss‚0.1058275936305209 &loss/total_loss‚40.256551549204325 1loss/FBt1/assignment_loss‚0.17820106367446442 0loss/SRqqt1/detection_loss‚2.623120869909014 /loss/FBt3/detection_loss‚3.0851897249619165 2loss/SRqqt1/assignment_loss‚1.3619451065148627 1loss/FBt3/assignment_loss‚0.23575270688161254 /loss/FRt1/detection_loss‚2.6913581502598687×3PS»(‚(:… ñòäÔ¶õ®y gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.392' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v|˜Òz¼(‚(:ï ñòäÔΕ„{ disk.sda.in‚ 7.3671875( proc.memory.availableMB‚ 336.51953125 disk.sda2.in‚0 disk.nvme0c0n1.in‚5.3125) proc.memory.percent‚17.47293472290039 cpu‚25.288755815570127 disk.dm-1.out‚ 468.45703125% disk.nvme0n1.out‚46754.6748046875 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625$ memory_percent‚97.94604778289795! proc.memory.rssMB‚ 2862.765625 network.sent‚ 2.6297072e+07! disk.rbd2.out‚287.99853515625 disk.rbd0.in‚0 disk.rbd4.out‚0 network.recv‚ 3.544397e+06 disk.rbd1.out‚ 4.96923828125 disk.dm-0.in‚5.3125" disk.dm-0.out‚46754.6748046875 disk.rbd3.in‚0) disk./.usagePercent‚73.73732399174811 disk.rbd1.in‚0' disk.nvme0c0n1.out‚46754.6748046875 disk.sda.out‚ 468.49609375 disk.sda3.in‚ 7.31640625 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda3.out‚ 468.45703125 proc.cpu.threads‚17 disk.nvme0n1.in‚5.3125 disk.rbd0.out‚0 disk.sda1.out‚0$ disk./.usageGB‚1098.739414215088 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚ 7.31640625 disk.drbd1777.out‚0Ý0%›½(‚(: €óäÔŒÿ»x gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.467' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.293400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õÝqo¾(‚(:ä €óäÔí²{ network.recv‚ 3.551497e+06 disk.rbd2.out‚288.1474609375 disk.sda1.in‚ 0.05078125 disk.sda3.in‚ 7.31640625! proc.memory.rssMB‚ 2862.765625 network.sent‚ 2.6405777e+07 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 proc.cpu.threads‚17) disk./.usagePercent‚73.73732322372551 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚469# disk.dm-0.out‚46755.25341796875$ memory_percent‚97.94912338256836 cpu‚25.288401615639305 disk.rbd0.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚46755.25341796875 disk.dm-0.in‚5.3125 disk.rbd4.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚46755.25341796875) proc.memory.percent‚17.47293472290039 disk.nvme0n1.in‚5.3125 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 468.9609375 disk.rbd1.out‚ 4.96923828125 disk.sda2.in‚0 disk.sda.in‚ 7.3671875$ disk./.usageGB‚1098.739402770996 disk.dm-1.in‚ 7.31640625 disk.dm-1.out‚ 468.9609375 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0& proc.memory.availableMB‚ 336.015625ü;A¿(‚(:… óäÔ“²x gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²™¦ñxÀ(‚(:í óäÔ•½Ï{ disk.drbd1777.in‚ 1.03515625% disk.nvme0n1.out‚46854.8818359375 network.sent‚ 2.6411713e+07) disk./.usagePercent‚73.73732245570291 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚289.7451171875 disk.sda.in‚ 7.3671875 disk.rbd0.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.47293472290039 proc.cpu.threads‚17 disk.dm-1.in‚ 7.31640625 cpu‚25.28804614765622 disk.sda3.out‚ 469.60546875 disk.sda2.in‚0 disk.rbd4.out‚0 disk.rbd3.out‚0$ memory_percent‚97.94754981994629! proc.memory.rssMB‚ 2862.765625 disk.rbd4.in‚0' disk.nvme0c0n1.out‚46854.8818359375" disk.dm-0.out‚46854.8818359375 disk.sda.out‚ 469.64453125 disk.rbd1.out‚ 4.96923828125 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 469.60546875% disk./.usageGB‚1098.7393913269043 disk.dm-0.in‚5.3125 disk.rbd0.out‚0' proc.memory.availableMB‚ 336.2734375 network.recv‚ 3.555933e+06 disk.sda3.in‚ 7.31640625 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0•£1|4Á(/ _wandbruntime‚10115  _runtime‚10115äåS%ÈÂ( & lr-AdamW/pg1‚4.419027555572854e-05 & lr-AdamW/pg2‚4.419027555572854e-05 trainer/global_step‚10449 % _timestamp‚1.7884266444402049e+09 _runtime‚10115.19013738 _step‚856vBvŽÍÃ(Ç & lr-AdamW/pg2‚4.419027555572854e-05 trainer/global_step‚10449 % _timestamp‚1.7884266444402049e+09 _runtime‚10115.19013738 _step‚856 & lr-AdamW/pg1‚4.419027555572854e-05ØÈ’(4Ä(/ _wandbruntime‚10115  _runtime‚10115vïäDð'Å(êO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19347601171384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚413823538564096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚411197216316724 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 464586422 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 $ _timestamp‚1.788426644441027e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚413823538564096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19740748115968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚433227438067712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 192703449 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _runtime‚10115.191220179 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19404145710080 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 192703447 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19347824820532 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚463795746621440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19740470905856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚411196975628216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚413823520561152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112738878 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 192703449 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 271908645 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 271882975 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚444054974565376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 183104952 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112738876 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19404145710080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚444054998505472 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚391849391496192 _step‚857 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70366730°ÿQßõ'Æ(ïO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19404145710080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚444054998505472 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚391849391496192 _step‚857 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70366730 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19347601171384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚413823538564096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚411197216316724 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 464586422 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 $ _timestamp‚1.788426644441027e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚413823538564096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19740748115968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚433227438067712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 192703449 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _runtime‚10115.191220179 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19404145710080 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 192703447 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19347824820532 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚463795746621440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19740470905856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚411196975628216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚413823520561152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112738878 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 192703449 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 271908645 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 271882975 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚444054974565376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 183104952 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112738876 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073ÙeS3O4Ç(/ _wandbruntime‚10116  _runtime‚10116¥±¾È(õM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚413861876508160 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 % _timestamp‚1.7884266452933733e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 192721168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚433267613706240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 464629692 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 271934222 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19742610939392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 SEDeviceStatsMonitor.on€ÝF="_train_batch_end/allocation.small_pool.allocated‚ 271934222 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚391885693954048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 _step‚858 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19405755201024 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 464655390 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚463838484295680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚444096168201728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 192721168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19405983417856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 464629692 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19349433570984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚463838779141120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19405755201024 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183121528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 271908526 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚413861858505216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚391885676914688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 183122168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70372297 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚413861876508160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 464655390 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19742340034048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚444096144261632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚10116.043696069N¾SC'É(úM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚10116.043696069 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚413861876508160 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 % _timestamp‚1.7884266452933733e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 192721168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚433267613706240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 464629692 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 271934222 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19742610939392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 271934222 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚391885693954048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 _step‚858 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19405755201024 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 464655390 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚463838484295680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚444096168201728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 192721168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19405983417856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 464629692 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19349433570984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚463838779141120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19405755201024 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183121528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 271908526 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚413861858505216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚391885676914688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 183122168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70372297 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚413861876508160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 464655390 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19742340034048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚444096144261632ÚåáVü4Ê(/ _wandbruntime‚10116  _runtime‚10116»:»l_Ë(Ù % _timestamp‚1.7884266452949388e+09 /loss/FRt2/detection_loss‚1.3535127013921737 2loss/SRqqt1/assignment_loss‚1.6738628544570768 2loss/SRqqt3/assignment_loss‚1.0940734986719247 2loss/SRqqt4/assignment_loss‚1.3546431728384711 1loss/SRqqt2/detection_loss‚0.9323084311688254 -loss/FBt1/detection_loss‚2.67399576994089 0loss/FRt4/assignment_loss‚3.4057922661304474 2loss/SRqqt2/assignment_loss‚0.7970707873203631 .loss/FRt3/detection_loss‚2.320721234787594 1loss/SRqqt1/detection_loss‚2.1034685758983387 epoch‚8 trainer/global_step‚10449 .loss/FRt1/detection_loss‚2.476099692285061 1loss/FBt2/assignment_loss‚0.10616759857392494 1loss/FBt3/assignment_loss‚0.04723036862976317 0loss/FBt4/assignment_loss‚0.5098398662594263 /loss/FRt4/detection_loss‚2.3196302753825315 0loss/FRt2/assignment_loss‚1.2015638720989228 0loss/FRt3/assignment_loss‚2.6452185660600662 1loss/SRqqt3/detection_loss‚1.9426424165029783 /loss/FBt2/detection_loss‚0.6180824665352702 _runtime‚10116.044766506 .loss/FBt3/detection_loss‚2.453023774283273 &loss/total_loss‚39.911621084829775 0loss/FRt1/assignment_loss‚2.8439883575552987 0loss/FBt1/assignment_loss‚0.3780897017281859 _step‚859 1loss/SRqqt4/detection_loss‚2.4430446976965126 /loss/FBt4/detection_loss‚2.2175501386324563.Û\ÖdÌ(Þ 1loss/SRqqt1/detection_loss‚2.1034685758983387 epoch‚8 trainer/global_step‚10449 .loss/FRt1/detection_loss‚2.476099692285061 1loss/FBt2/assignment_loss‚0.10616759857392494 1loss/FBt3/assignment_loss‚0.04723036862976317 0loss/FBt4/assignment_loss‚0.5098398662594263 /loss/FRt4/detection_loss‚2.3196302753825315 0loss/FRt2/assignment_loss‚1.2015638720989228 0loss/FRt3/assignment_loss‚2.6452185660600662 1loss/SRqqt3/detection_loss‚1.9426424165029783 /loss/FBt2/detection_loss‚0.6180824665352702 _runtime‚10116.044766506 .loss/FBt3/detection_loss‚2.453023774283273 &loss/total_loss‚39.911621084829775 0loss/FRt1/assignment_loss‚2.8439883575552987 0loss/FBt1/assignment_loss‚0.3780897017281859 _step‚859 1loss/SRqqt4/detection_loss‚2.4430446976965126 /loss/FBt4/detection_loss‚2.2175501386324563 % _timestamp‚1.7884266452949388e+09 /loss/FRt2/detection_loss‚1.3535127013921737 2loss/SRqqt1/assignment_loss‚1.6738628544570768 2loss/SRqqt3/assignment_loss‚1.0940734986719247 2loss/SRqqt4/assignment_loss‚1.3546431728384711 1loss/SRqqt2/detection_loss‚0.9323084311688254 -loss/FBt1/detection_loss‚2.67399576994089 0loss/FRt4/assignment_loss‚3.4057922661304474 2loss/SRqqt2/assignment_loss‚0.7970707873203631 .loss/FRt3/detection_loss‚2.320721234787594Û_,áŽÍ(‚(:ƒ žóäÔ„×¾x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.57' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<•›tÎ(‚(:é žóäÔœÓè{ disk.sda.in‚ 7.3671875 disk.sda3.out‚ 469.8828125" disk.dm-0.out‚46954.0068359375 disk.sda2.in‚0 disk.dm-1.in‚ 7.31640625 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd1.out‚ 4.96923828125 proc.cpu.threads‚17) disk./.usagePercent‚73.73732245570291 disk.dm-0.in‚5.3125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚46954.0068359375 disk.sda1.out‚0 disk.sda3.in‚ 7.31640625 cpu‚25.28769343652398 network.sent‚ 2.6417675e+07% disk./.usageGB‚1098.7393913269043 disk.rbd2.out‚289.8271484375 disk.drbd1777.in‚ 1.03515625( proc.memory.availableMB‚ 336.23046875 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 469.921875# memory_percent‚97.9478120803833! proc.memory.rssMB‚ 2862.765625) proc.memory.percent‚17.47293472290039 network.recv‚ 3.560018e+06% disk.nvme0n1.out‚46954.0068359375 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.dm-1.out‚ 469.8828125 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0ã:­›Ï(‚(: ­óäÔ¡áŸx gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.156' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.031200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§Ö¼‰}Ð(‚(:ò ­óäÔ…ž’|& disk.nvme0n1.out‚47053.29736328125 network.recv‚ 3.567473e+06 disk.dm-1.out‚ 470.41015625 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0! proc.memory.rssMB‚ 2862.765625 disk.dm-1.in‚ 7.31640625 disk.rbd4.out‚0 disk.sda.out‚ 470.44921875) proc.memory.percent‚17.47293472290039 cpu‚25.287341058934338 proc.cpu.threads‚17) disk./.usagePercent‚73.73732245570291# disk.dm-0.out‚47053.29736328125 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 5.05517578125 disk.dm-0.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.rbd1.in‚0% disk./.usageGB‚1098.7393913269043 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚47053.29736328125 disk.nvme0n1.in‚5.3125$ memory_percent‚97.94924259185791 disk.sda3.in‚ 7.31640625( proc.memory.availableMB‚ 335.99609375 network.sent‚ 2.6527024e+07 disk.sda2.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd2.out‚291.3662109375 disk.sda.in‚ 7.3671875 disk.sda3.out‚ 470.41015625Ut‡EšÑ(‚(: ¼óäԘÒx gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.496' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.899199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02¬,tÒ(‚(:é ¼óäÔÎÞìz disk.rbd3.in‚0 disk.sda.in‚ 7.3671875 disk.rbd1.in‚0 network.recv‚ 3.571124e+06 disk.sda1.out‚0 disk.rbd1.out‚ 5.11767578125 disk.rbd4.in‚0 disk.dm-0.in‚5.3125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚47053.59814453125 disk.rbd3.out‚0 disk.dm-1.in‚ 7.31640625 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625$ memory_percent‚97.94778823852539& proc.memory.availableMB‚ 336.234375 network.sent‚ 2.6532399e+07) disk./.usagePercent‚73.73732373574059% disk./.usageGB‚1098.7394104003906 disk.sda3.out‚ 470.90625& disk.nvme0n1.out‚47053.59814453125 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 470.9453125) proc.memory.percent‚17.47293472290039 proc.cpu.threads‚17 disk.sda3.in‚ 7.31640625 disk.nvme0n1.in‚5.3125 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125! proc.memory.rssMB‚ 2862.765625 cpu‚25.28699801097508# disk.dm-0.out‚47053.59814453125 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 470.90625! disk.rbd2.out‚291.58154296875$˜ñ4Ó(/ _wandbruntime‚10160  _runtime‚10160'Ó‘ËÔ(Å _step‚860 ' lr-AdamW/pg1‚4.1585240860532436e-05 ' lr-AdamW/pg2‚4.1585240860532436e-05 trainer/global_step‚10499 % _timestamp‚1.7884266896876884e+09 _runtime‚10160.437645948âØÞóÐÕ(Ê trainer/global_step‚10499 % _timestamp‚1.7884266896876884e+09 _runtime‚10160.437645948 _step‚860 ' lr-AdamW/pg1‚4.1585240860532436e-05 ' lr-AdamW/pg2‚4.1585240860532436e-05Üy©?¥4Ö(/ _wandbruntime‚10160  _runtime‚10160å%Ï1×(ëO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚413103972780732 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚393664497349632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚643 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚415740417764352 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19495816643584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚415740417764352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚10499 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 _step‚861 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚446113483318272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚415740435767296 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 193589397 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70692369 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 183948354 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚435236480614400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 193589399 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19439458391740 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚641 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446113459378176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚393664514388992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 273163877 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 466778946 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70691728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19834200627712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 273189547 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19496044847104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚435236234407936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465947683945984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19833925514752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚435236234407936 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113256626 RDDevic·à´#ÀeStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚415740435767296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 % _timestamp‚1.7884266896886382e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 466753274 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465947683945984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19833925514752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚435236234407936 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113256626 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚415740435767296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 % _timestamp‚1.7884266896886382e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 466753274 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70692369 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 183948354 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚435236480614400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 193589399 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19439458391740 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚641 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446113459378176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚393664514388992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 273163877Ý–I‹±4Ù(/ _wandbruntime‚10161  _runtime‚10161X—ú&Ú(ôM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19497882554880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 284359680 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 466822242 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚435276410046464 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113266983 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19441067142192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚435276656266240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚415778773711360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚415778755708416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 _step‚862 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚435276656266240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚465990735948288 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 260419584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚413142107638396 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 193607116 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 193607118 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446154653014528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19441290791548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 193607116 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446154629074432 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 466796544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 273189428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70698632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19836082933760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚393700816846848 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚10499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183964987 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚393700799807488 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚415778755708416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19497654338048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 273215124 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚413141866949680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 193607118 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 183965615 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 466822242 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4157787737113604ôæ*ùÛ(ùM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 193607116 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446154629074432 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 466796544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 273189428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70698632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19836082933760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚393700816846848 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚10499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183964987 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚393700799807488 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚415778755708416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19497654338048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 273215124 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚413141866949680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 193607118 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 183965615 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 466822242 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚415778773711360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19497882554880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 284359680 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 466822242 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚435276410046464 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113266983 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19441067142192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚435276656266240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚415778773711360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚415778755708416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 _step‚862 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚435276656266240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚465990735948288 >6DeviceStatsMonitor.on_train_batch_end/num_d2¦Œevice_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 260419584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚413142107638396 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 193607116 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 193607118 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446154653014528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19441290791548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ÞËx¬å4Ü(/ _wandbruntime‚10161  _runtime‚10161×öA_Ý(Ù $loss/total_loss‚42.7614834608221 trainer/global_step‚10499 /loss/FRt2/detection_loss‚1.5988686098175686 0loss/FRt3/assignment_loss‚2.6127107196384007 2loss/SRqqt3/assignment_loss‚1.7610108661016024 1loss/FBt1/assignment_loss‚0.27515392106531067 1loss/FBt2/assignment_loss‚0.09341318247449064 _runtime‚10161.303059251 1loss/SRqqt1/detection_loss‚2.1363061652297066 /loss/FBt4/detection_loss‚2.7263227701187134 _step‚863 $ _timestamp‚1.788426690553232e+09 .loss/FRt3/detection_loss‚2.484673649072647 /loss/FRt4/detection_loss‚2.1743212040257514 2loss/SRqqt2/assignment_loss‚1.0967409103729968 0loss/FBt3/assignment_loss‚1.2539739406201988 .loss/FBt1/detection_loss‚1.941177360713482 0loss/FRt2/assignment_loss‚1.6279011729151704 0loss/FRt4/assignment_loss‚3.2552231489083705 2loss/SRqqt4/assignment_loss‚1.5489395821636374 /loss/FBt3/detection_loss‚3.5130868554115295 epoch‚8 /loss/FRt1/detection_loss‚2.6097918916493654 0loss/FRt1/assignment_loss‚2.9814001669486365 1loss/FBt4/assignment_loss‚0.36171935595727217 1loss/SRqqt2/detection_loss‚1.0468831516928592 1loss/SRqqt1/assignment_loss‚1.645118771830485 1loss/SRqqt3/detection_loss‚1.5996213390638954 1loss/SRqqt4/detection_loss‚1.7522443484176289 /loss/FBt2/detection_loss‚0.6648803766123778DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437245030748160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚417657332970496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395479637281792 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 274444779 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚437245030748160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 194475349 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 274470449 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19927831493120 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19587715780608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19531091962948 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚468099506862592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417657314967552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 184793274 _runtime‚10205.657400249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113774378 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚448171944190976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417657332970496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415010729244740 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 274444779 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19587715780608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚437245276954624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19927562671616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71019530 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415010488556232 $ _timestamp‚1.788426734907261e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 468945798 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚468099799624192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 468920126 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19587943984128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 184793908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚632 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437245030748160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚417657332970496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395479637281792 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 274444779 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚437245030748160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 194475349 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 274470449 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19927831493120 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19587715780608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19531091962948 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚468099506862592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417657314967552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 184793274 _runtime‚10205.657400249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113774378 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚448171944190976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417657332970496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415010729244740 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 274444779 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19587715780608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624áeÃg4ë(/ _wandbruntime‚10206  _runtime‚10206±é-_ ì(õM P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 194493068 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 274470330 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113784731 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚415048864102404 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417695652911616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395515939739648 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19532924362756 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 274496026 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417695670914560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 468989094 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚468142563088896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19929724301312 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 194493068 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113784733 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚468142862128640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19929449201664 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _step‚866 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 274496026 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚415048623413688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71025898 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19589781691904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184809976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚448213137827328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19532700713400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19589553475072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 194493066 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395515922700288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417695670914560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 468989094 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19589553475072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71025245 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 trainer/global_step‚10549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚437285452606464]ô'í(úM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚415048623413688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71025898 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19589781691904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184809976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚448213137827328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19532700713400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19589553475072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 194493066 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395515922700288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417695670914560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 468989094 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19589553475072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71025245 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 trainer/global_step‚10549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚437285452606464 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 194493068 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 274470330 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113784731 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚415048864102404 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417695652911616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395515939739648 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19532924362756 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 274496026 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417695670914560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 468989094 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚468142563088896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19929724301312 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 194493068 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113784733 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚468142862128640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19929449201664 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 _step‚866 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 274496026 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655â8ò734î(/ _wandbruntime‚10206  _runtime‚10206Ÿ×[ï(Õ .loss/FRt4/detection_loss‚2.267170148430153 1loss/SRqqt1/detection_loss‚2.2560838950531825 /loss/FBt4/detection_loss‚2.3935101449489595 _runtime‚10206.512194069 /loss/FRt4/assignment_loss‚2.863846031824748 /loss/FRt3/detection_loss‚2.6054080165014035 epoch‚8 0loss/FBt1/assignment_loss‚0.2585789660094709 1loss/FBt2/assignment_loss‚0.15004705505014224 2loss/SRqqt2/assignment_loss‚0.9722879785630438 2loss/SRqqt3/assignment_loss‚1.3824307377958163 /loss/FRt2/detection_loss‚1.5507390263785676 0loss/SRqqt4/detection_loss‚2.217250731004321 .loss/FBt3/detection_loss‚3.121628451347351 trainer/global_step‚10549 2loss/SRqqt1/assignment_loss‚1.5134553359377951 .loss/FBt1/detection_loss‚2.299836536248525 /loss/FBt2/detection_loss‚0.6636347083395786 /loss/FRt1/assignment_loss‚2.812657329298201 0loss/FRt2/assignment_loss‚1.2127818959515269 1loss/FBt4/assignment_loss‚0.21970832950391922 1loss/SRqqt2/detection_loss‚0.8942236252542999 0loss/SRqqt3/detection_loss‚1.867823689160022 &loss/total_loss‚40.366950983820004 _step‚867 $ _timestamp‚1.788426735762373e+09 2loss/SRqqt4/assignment_loss‚1.7532974152785281 0loss/FBt3/assignment_loss‚0.0661138498224318 /loss/FRt1/detection_loss‚2.4110958331397603 /loss/FRt3/assignment_loss‚2.613341252978255x)0`ð(Ú /loss/FRt3/assignment_loss‚2.613341252978255 .loss/FRt4/detection_loss‚2.267170148430153 1loss/SRqqt1/detection_loss‚2.2560838950531825 /loss/FBt4/detection_loss‚2.3935101449489595 _runtime‚10206.512194069 /loss/FRt4/assignment_loss‚2.863846031824748 /loss/FRt3/detection_loss‚2.6054080165014035 epoch‚8 0loss/FBt1/assignment_loss‚0.2585789660094709 1loss/FBt2/assignment_loss‚0.15004705505014224 2loss/SRqqt2/assignment_loss‚0.9722879785630438 2loss/SRqqt3/assignment_loss‚1.3824307377958163 /loss/FRt2/detection_loss‚1.5507390263785676 0loss/SRqqt4/detection_loss‚2.217250731004321 .loss/FBt3/detection_loss‚3.121628451347351 trainer/global_step‚10549 2loss/SRqqt1/assignment_loss‚1.5134553359377951 .loss/FBt1/detection_loss‚2.299836536248525 /loss/FBt2/detection_loss‚0.6636347083395786 /loss/FRt1/assignment_loss‚2.812657329298201 0loss/FRt2/assignment_loss‚1.2127818959515269 1loss/FBt4/assignment_loss‚0.21970832950391922 1loss/SRqqt2/detection_loss‚0.8942236252542999 0loss/SRqqt3/detection_loss‚1.867823689160022 &loss/total_loss‚40.366950983820004 _step‚867 $ _timestamp‚1.788426735762373e+09 2loss/SRqqt4/assignment_loss‚1.7532974152785281 0loss/FBt3/assignment_loss‚0.0661138498224318 /loss/FRt1/detection_loss‚2.4110958331397603ã2é,ñ(‚(:… øóäԀܜx gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V_Îoò(‚(:ä øóäÔÔøÓ{ disk.rbd2.out‚ 296.58203125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚ 7.31640625) disk./.usagePercent‚73.73732962391387 disk.sda2.in‚0$ memory_percent‚97.94840812683105( disk.nvme0c0n1.out‚47351.69091796875 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 473.1015625 disk.sda3.in‚ 7.31640625# disk.dm-0.out‚47351.69091796875 disk.drbd1777.out‚0 disk.rbd4.out‚0& disk.nvme0n1.out‚47351.69091796875) proc.memory.percent‚17.47293472290039% disk./.usageGB‚1098.7394981384277 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda3.out‚473.0625 disk.rbd1.out‚ 5.212890625 cpu‚25.285605487541886 disk.rbd1.in‚0 disk.dm-1.out‚473.0625' proc.memory.availableMB‚ 336.1328125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.in‚ 7.3671875! proc.memory.rssMB‚ 2862.765625 network.sent‚ 2.6658764e+07 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 network.recv‚ 3.590555e+06 disk.rbd2.in‚0xÚŽ ó(‚(:… ‡ôäÔ‹­üx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«dâ*oô(‚(:ä ‡ôäÔØàÉ{ disk.dm-0.in‚5.3125# disk.dm-0.out‚47450.96728515625 disk.dm-1.in‚ 7.31640625 disk.sda3.in‚ 7.31640625( disk.nvme0c0n1.out‚47450.96728515625 network.recv‚ 3.597722e+06 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 disk.dm-1.out‚ 473.78125 disk.drbd1777.in‚ 1.03515625% disk./.usageGB‚1098.7394981384277 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 473.78125 disk.rbd1.in‚0) proc.memory.percent‚17.47293472290039) disk./.usagePercent‚73.73732962391387 disk.rbd4.in‚0$ memory_percent‚97.94836044311523& proc.memory.availableMB‚ 336.140625 network.sent‚ 2.6767589e+07 disk.rbd0.out‚0 disk.sda1.out‚0 cpu‚25.28523694153024 disk.rbd2.out‚ 296.60546875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625 disk.rbd4.out‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚47450.96728515625 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.3671875 disk.sda.out‚ 473.8203125 disk.rbd1.out‚ 5.212890625! proc.memory.rssMB‚ 2862.765625…ë„`›õ(‚(: –ôäÔ¹ŒÄx gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.969' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.593799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰Äïzö(‚(:ï –ôäÔÒ¯Ôz network.recv‚ 3.602113e+06 disk.rbd4.out‚0 disk.rbd1.out‚ 5.27587890625 disk.dm-1.in‚ 7.31640625 disk.rbd3.in‚0) proc.memory.percent‚17.47293472290039 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 474.16796875$ memory_percent‚97.94690608978271! proc.memory.rssMB‚ 2862.765625 proc.cpu.threads‚17 disk.rbd0.in‚0( proc.memory.availableMB‚ 336.37890625( disk.nvme0c0n1.out‚47550.21044921875 disk.sda3.out‚ 474.16796875 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚47550.21044921875% disk./.usageGB‚1098.7396507263184 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 cpu‚25.284898043622047 disk.rbd2.out‚ 298.3203125 disk.rbd4.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚47550.21044921875) disk./.usagePercent‚73.73733986421524 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.3671875 disk.sda.out‚ 474.20703125 disk.sda3.in‚ 7.31640625 disk.dm-0.in‚5.3125 network.sent‚ 2.6773512e+07û "4÷(/ _wandbruntime‚10250  _runtime‚10250úÂwÉø(à & lr-AdamW/pg2‚3.660050284634571e-05 trainer/global_step‚10599 % _timestamp‚1.7884267801975873e+09 _runtime‚10250.947490627 _step‚868 & lr-AdamW/pg1‚3.660050284634571e-05›Ö`ÑÎù(È % _timestamp‚1.7884267801975873e+09 _runtime‚10250.947490627 _step‚868 & lr-AdamW/pg1‚3.660050284634571e-05 & lr-AdamW/pg2‚3.660050284634571e-05 trainer/global_step‚10599ä{’z‘4ú(/ _wandbruntime‚10250  _runtime‚10250Û¥ÕEóû(ëO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195361297 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19679614917632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚416917245020240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19679614917632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚419574212170752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚439254073294848 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114292126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71343368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71342700 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚397294760174592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚419574230173696 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185634826 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450230452943872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.Î% þ all.allocated‚561 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439253827088384 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 275751351 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20021111765504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19622501885008 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚10250.948377077 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19679843121152 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262BŒ±þö'ü(ðO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 275725681 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419574212170752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20021391072768 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚439254073294848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 275725681 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚470251844016640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚470251540769280 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 471086978 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 275751351 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚397294743135232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19622725534156 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚668 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450230429003776 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 471112650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19679614917632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚416917245020240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19679614917632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚419574212170752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚439254073294848 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114292126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71343368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71342700 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚397294760174592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚419574230173696 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185634826 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450230452943872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439253827088384 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 275751351 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20021111765504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19622501885008 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚10250.948377077 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19679843121152 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195361297 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419612550114816 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 185652051 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20023272275456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 195379016 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19624334284608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19681452612096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 471130248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚10599 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 195379018 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419612568117760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19681680828928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚470294604264448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚416955620566412 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 $ _timestamp‚1.788426781054522e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114302483 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185651411 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450271622640128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚416955379877696 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 471155946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 275776928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 195379018 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114302481 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20023005564416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19624557933964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚439294248946688 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 275751232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 _runtime‚10251.805210751 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 275751232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 275776928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚450271598700032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561PCšR»$ÿ(ùM OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 195379016 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19624334284608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19681452612096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 471130248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚10599 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 195379018 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419612568117760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19681680828928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚470294604264448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚416955620566412 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 $ _timestamp‚1.788426781054522e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114302483 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185651411 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450271622640128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚416955379877696 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 471155946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 275776928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 195379018 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114302481 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20023005564416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19624557933964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚439294248946688 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 275751232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 _runtime‚10251.805210751 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 275751232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 275776928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚450271598700032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19681680828928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419612550114816 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.aló@XÂDlocated‚ 185652051 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20023272275456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698æÙ_<4€)/ _wandbruntime‚10251  _runtime‚10251åZ)Ô % _timestamp‚1.7884267810564084e+09 /loss/FRt3/assignment_loss‚3.395541548728943 /loss/FRt4/assignment_loss‚2.517496141227516 1loss/SRqqt1/assignment_loss‚1.691419119405192 1loss/FBt1/assignment_loss‚0.04546617459921739 /loss/SRqqt1/detection_loss‚2.04149961125019 /loss/FBt2/detection_loss‚0.5339140669753154 /loss/FBt3/detection_loss‚2.0452719005671414 0loss/FRt1/assignment_loss‚2.6778519418504505 0loss/FRt2/assignment_loss‚1.5523201168752183 2loss/SRqqt3/assignment_loss‚1.0941527641245297 .loss/FRt1/detection_loss‚2.427413428616193 1loss/SRqqt4/detection_loss‚1.6608459003946998 _step‚871 2loss/SRqqt2/assignment_loss‚1.0617851764666584 1loss/FBt4/assignment_loss‚0.11533477746310052 %loss/total_loss‚39.78133202145253 2loss/SRqqt4/assignment_loss‚1.5250258260152556 1loss/FBt2/assignment_loss‚0.18241526823153545 /loss/SRqqt3/detection_loss‚2.11849562057427 .loss/FBt4/detection_loss‚2.460279800675132 trainer/global_step‚10599 .loss/FRt3/detection_loss‚2.934505753006254 /loss/FBt1/detection_loss‚2.8387795289357505 epoch‚8 1loss/FBt3/assignment_loss‚0.27696450138252904 /loss/FRt2/detection_loss‚1.4668542275607002 _runtime‚10251.806211837 .loss/FRt4/detection_loss‚2.191067563614227 1loss/SRqqt2/detection_loss‚0.9266312629125029+?B&_‚)Ù _step‚871 2loss/SRqqt2/assignment_loss‚1.0617851764666584 1loss/FBt4/assignment_loss‚0.11533477746310052 %loss/total_loss‚39.78133202145253 2loss/SRqqt4/assignment_loss‚1.5250258260152556 1loss/FBt2/assignment_loss‚0.18241526823153545 /loss/SRqqt3/detection_loss‚2.11849562057427 .loss/FBt4/detection_loss‚2.460279800675132 trainer/global_step‚10599 .loss/FRt3/detection_loss‚2.934505753006254 /loss/FBt1/detection_loss‚2.8387795289357505 epoch‚8 1loss/FBt3/assignment_loss‚0.27696450138252904 /loss/FRt2/detection_loss‚1.4668542275607002 _runtime‚10251.806211837 .loss/FRt4/detection_loss‚2.191067563614227 1loss/SRqqt2/detection_loss‚0.9266312629125029 % _timestamp‚1.7884267810564084e+09 /loss/FRt3/assignment_loss‚3.395541548728943 /loss/FRt4/assignment_loss‚2.517496141227516 1loss/SRqqt1/assignment_loss‚1.691419119405192 1loss/FBt1/assignment_loss‚0.04546617459921739 /loss/SRqqt1/detection_loss‚2.04149961125019 /loss/FBt2/detection_loss‚0.5339140669753154 /loss/FBt3/detection_loss‚2.0452719005671414 0loss/FRt1/assignment_loss‚2.6778519418504505 0loss/FRt2/assignment_loss‚1.5523201168752183 2loss/SRqqt3/assignment_loss‚1.0941527641245297 .loss/FRt1/detection_loss‚2.427413428616193 1loss/SRqqt4/detection_loss‚1.6608459003946998ç™9ôƒ)‚(:… ¥ôäÔÖâx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØUmt„)‚(:é ¥ôäÔã¡—{ disk.sda.in‚ 7.3671875 disk.dm-1.in‚ 7.31640625 cpu‚25.284552849083823% disk.nvme0n1.out‚47551.0322265625 disk.dm-1.out‚ 474.68359375 network.recv‚ 3.606815e+06 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚97.95026779174805) disk./.usagePercent‚73.73734012022277 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda.out‚ 474.72265625 disk.rbd1.in‚0 disk.dm-0.in‚5.3125& proc.memory.availableMB‚ 335.828125% disk./.usageGB‚1098.7396545410156 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.rbd2.in‚0) proc.memory.percent‚17.47293472290039 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625' disk.nvme0c0n1.out‚47551.0322265625 disk.rbd1.out‚ 5.27587890625" disk.dm-0.out‚47551.0322265625 disk.sda2.out‚ 0.0390625 disk.sda3.out‚ 474.68359375 network.sent‚ 2.6779851e+07 disk.sda1.in‚ 0.05078125 disk.sda3.in‚ 7.31640625 disk.rbd2.out‚ 299.921875! proc.memory.rssMB‚ 2862.765625a²žŽ…)‚(:„ ´ôäÔ›Ýx gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚94.971' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á›Èóu†)‚(:ê ´ôäÔõäâ{ network.sent‚ 2.6887975e+07 disk.sda.out‚ 475.12890625 disk.dm-1.out‚ 475.08984375 proc.cpu.threads‚17 disk.dm-1.in‚ 7.31640625 disk.rbd2.out‚ 300.06640625' disk.nvme0c0n1.out‚47650.1650390625' proc.memory.availableMB‚ 335.9453125 cpu‚25.28418407089392 disk.nvme0n1.in‚5.3125 disk.sda3.out‚ 475.08984375" disk.dm-0.out‚47650.1650390625 disk.sda2.out‚ 0.0390625 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0$ memory_percent‚97.94955253601074! proc.memory.rssMB‚ 2862.765625 disk.rbd1.out‚ 5.27587890625% disk.nvme0n1.out‚47650.1650390625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0% disk./.usageGB‚1098.7396545410156 disk.rbd1.in‚0 disk.drbd1777.out‚0 network.recv‚ 3.61313e+06) disk./.usagePercent‚73.73734012022277 disk.sda.in‚ 7.3671875 disk.sda3.in‚ 7.31640625 disk.dm-0.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd0.in‚0) proc.memory.percent‚17.47293472290039•ÿ8›‡)‚(: ÃôäÔꯄx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.389' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.677799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oeKyˆ)‚(:î ÃôäÔñγ{ disk.rbd2.in‚0 network.sent‚ 2.6894038e+07 disk.rbd3.in‚0 disk.sda3.in‚ 7.31640625& disk.nvme0n1.out‚47749.28759765625 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0% disk./.usageGB‚1098.7396545410156 disk.dm-1.out‚ 475.69140625 disk.rbd0.in‚0 disk.drbd1777.out‚0# memory_percent‚97.9511022567749 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125) proc.memory.percent‚17.47293472290039 network.recv‚ 3.617446e+06) disk./.usagePercent‚73.73734012022277 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.sda3.out‚ 475.69140625( proc.memory.availableMB‚ 335.69140625 cpu‚25.283821231475667 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚ 301.6640625( disk.nvme0c0n1.out‚47749.28759765625 disk.rbd3.out‚0 disk.dm-0.in‚5.3125 disk.sda.out‚ 475.73046875 disk.dm-1.in‚ 7.31640625# disk.dm-0.out‚47749.28759765625 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.sda.in‚ 7.3671875 disk.rbd1.out‚ 5.27587890625 disk.sda2.in‚0! proc.memory.rssMB‚ 2862.765625`Ƽ¯4‰)/ _wandbruntime‚10296  _runtime‚10296· KÉŠ)à _step‚872 & lr-AdamW/pg1‚3.422183833928585e-05 & lr-AdamW/pg2‚3.422183833928585e-05 trainer/global_step‚10649 % _timestamp‚1.7884268255850847e+09 _runtime‚10296.335045343jƒ›‡Î‹)È _step‚872 & lr-AdamW/pg1‚3.422183833928585e-05 & lr-AdamW/pg2‚3.422183833928585e-05 trainer/global_step‚10649 % _timestamp‚1.7884268255850847e+09 _runtime‚10296.335045343è=÷›û4Œ)/ _wandbruntime‚10296  _runtime‚10296ËjÓ ñ')ëO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 % _timestamp‚1.7884268255858748e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114809876 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399109883067392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚441262623428608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚418824242172756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 196247249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚10296.336047049 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 196247247 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 trainer/global_step‚10649 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚421491127376896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 196247249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚418824001484248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚441262869635072 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19771742258176 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 277032253 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚421491109373952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71667490 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚441262623428608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚472403536110080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20114622293504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19714135456216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚452288937756672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 196247247 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71666843 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚472403828871680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19771514054656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 186476719 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19714359105364 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚441262869635072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 186477368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649Ó½Xö'Ž)ðO QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 196247247 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 trainer/global_step‚10649 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚421491127376896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 196247249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚418824001484248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚441262869635072 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19771742258176 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 277032253 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚421491109373952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71667490 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚441262623428608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚472403536110080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20114622293504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19714135456216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚452288937756672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 196247247 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71666843 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚472403828871680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19771514054656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 186476719 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19714359105364 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚441262869635072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 186477368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 % _timestamp‚1.7884268255858748e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114809876 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399109883067392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚441262623428608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚418824242172756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 196247249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚10296.336047049éÁVt–4)/ _wandbruntime‚10297  _runtime‚10297å1ÁŒ)õM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 277057830 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 trainer/global_step‚10649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚421529465320960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 277032134 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19716191505172 _runtime‚10297.199969465 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186493152 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186493779 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 196264968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚441302799067136 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚874 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 473297100 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71673546 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚472446573554688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚452330083512832 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 196264966 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 277032134 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚452330107452928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399146168485888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19773351749120 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 277057830 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚441303045286912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚421529465320960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚472446870497280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 % _timestamp‚1.7884268264492934e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 473297100 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19773579965952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 473322798 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 473322798 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚09éw¡'‘)úM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 473322798 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 277057830 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 trainer/global_step‚10649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚421529465320960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 277032134 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19716191505172 _runtime‚10297.199969465 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186493152 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186493779 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 196264968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚441302799067136 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚874 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 473297100 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71673546 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚472446573554688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚452330083512832 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 196264966 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 277032134 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚452330107452928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399146168485888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19773351749120 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 277057830 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚441303045286912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚421529465320960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚472446870497280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 % _timestamp‚1.7884268264492934e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 473297100 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19773579965952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 473322798 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0êºåå04’)/ _wandbruntime‚10297  _runtime‚102973Zm\“)Ö -loss/FRt2/detection_loss‚1.71273781971799 .loss/FRt4/detection_loss‚2.569518541401609 1loss/SRqqt3/detection_loss‚1.7621056499833962 /loss/FBt2/detection_loss‚0.5052181982105916 trainer/global_step‚10649 0loss/FBt1/assignment_loss‚0.3754657498959984 1loss/FBt2/assignment_loss‚0.08148636680654628 1loss/SRqqt2/detection_loss‚0.8087422161454886 .loss/FBt3/detection_loss‚2.898440048098564 0loss/FRt4/assignment_loss‚3.0592015816105738 2loss/SRqqt3/assignment_loss‚1.5657543249580326 /loss/FBt1/detection_loss‚2.0421929955482483 %loss/total_loss‚39.99571069152913 0loss/FRt2/assignment_loss‚1.4006554590331184 2loss/SRqqt1/assignment_loss‚1.4437269101088697 /loss/FBt4/detection_loss‚2.3991383810838065 % _timestamp‚1.7884268264511778e+09 /loss/FRt1/assignment_loss‚2.350714189250295 1loss/FBt3/assignment_loss‚0.00883137242635712 /loss/FRt3/detection_loss‚2.5190522581972976 0loss/FBt4/assignment_loss‚0.6221195002241681 0loss/SRqqt1/detection_loss‚1.995676701041785 epoch‚8 _runtime‚10297.201022187 _step‚875 2loss/SRqqt2/assignment_loss‚0.8579353974069613 2loss/SRqqt4/assignment_loss‚1.4543430960696677 1loss/SRqqt4/detection_loss‚1.9725596538704375 .loss/FRt1/detection_loss‚2.502305202731272 0loss/FRt3/assignment_loss‚3.0877890777080617OÙoa”)Û .loss/FBt3/detection_loss‚2.898440048098564 0loss/FRt4/assignment_loss‚3.0592015816105738 2loss/SRqqt3/assignment_loss‚1.5657543249580326 /loss/FBt1/detection_loss‚2.0421929955482483 %loss/total_loss‚39.99571069152913 0loss/FRt2/assignment_loss‚1.4006554590331184 2loss/SRqqt1/assignment_loss‚1.4437269101088697 /loss/FBt4/detection_loss‚2.3991383810838065 % _timestamp‚1.7884268264511778e+09 /loss/FRt1/assignment_loss‚2.350714189250295 1loss/FBt3/assignment_loss‚0.00883137242635712 /loss/FRt3/detection_loss‚2.5190522581972976 0loss/FBt4/assignment_loss‚0.6221195002241681 0loss/SRqqt1/detection_loss‚1.995676701041785 epoch‚8 _runtime‚10297.201022187 _step‚875 2loss/SRqqt2/assignment_loss‚0.8579353974069613 2loss/SRqqt4/assignment_loss‚1.4543430960696677 1loss/SRqqt4/detection_loss‚1.9725596538704375 .loss/FRt1/detection_loss‚2.502305202731272 0loss/FRt3/assignment_loss‚3.0877890777080617 -loss/FRt2/detection_loss‚1.71273781971799 .loss/FRt4/detection_loss‚2.569518541401609 1loss/SRqqt3/detection_loss‚1.7621056499833962 /loss/FBt2/detection_loss‚0.5052181982105916 trainer/global_step‚10649 0loss/FBt1/assignment_loss‚0.3754657498959984 1loss/FBt2/assignment_loss‚0.08148636680654628 1loss/SRqqt2/detection_loss‚0.8087422161454886ë)°Q¹•)‚(:… ÒôäÔ†þx gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚108.357' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6714 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùåÝŒz–)‚(:ï ÒôäÔµçå{ disk.rbd3.in‚0 disk.sda3.in‚ 7.31640625 disk.rbd2.out‚ 301.8046875 network.recv‚ 3.621352e+06) disk./.usagePercent‚73.73737698530772 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 network.sent‚ 2.6899861e+07% disk./.usageGB‚1098.7402038574219 disk.sda3.out‚ 476.16015625 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0( proc.memory.availableMB‚ 335.53515625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.in‚ 7.3671875 disk.dm-1.in‚ 7.31640625( disk.nvme0c0n1.out‚47749.64697265625 cpu‚25.283479745067385& disk.nvme0n1.out‚47749.64697265625 disk.dm-0.in‚5.3125# disk.dm-0.out‚47749.64697265625 disk.drbd1777.out‚0 disk.sda.out‚ 476.19921875 disk.rbd1.in‚0 disk.drbd1777.in‚ 1.03515625! proc.memory.rssMB‚ 2862.765625) proc.memory.percent‚17.47293472290039 disk.dm-1.out‚ 476.16015625 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.rbd1.out‚ 5.27587890625$ memory_percent‚97.95205593109131 disk.rbd2.in‚0óH¶Q—)‚(:… áôäÔšè”x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.682' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1364 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ*¦su˜)‚(:ê áôäÔž¡èz disk.rbd4.in‚0 disk.sda.in‚ 7.3671875 disk.nvme0n1.in‚5.3125 disk.rbd0.out‚0% proc.memory.availableMB‚ 335.28125 disk.sda1.in‚ 0.05078125 disk.sda3.in‚ 7.31640625 disk.rbd1.in‚0 disk.rbd1.out‚ 5.27587890625& disk.nvme0n1.out‚47848.90478515625) disk./.usagePercent‚73.73737698530772% disk./.usageGB‚1098.7402038574219 disk.sda1.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚ 7.31640625 disk.rbd2.out‚ 303.40625$ memory_percent‚97.95360565185547! proc.memory.rssMB‚ 2862.765625 disk.sda.out‚ 476.72265625# disk.dm-0.out‚47848.90478515625) proc.memory.percent‚17.47293472290039 network.recv‚ 3.629159e+06 disk.drbd1777.in‚ 1.03515625 disk.rbd0.in‚0 network.sent‚ 2.7008943e+07 disk.sda3.out‚ 476.68359375 disk.sda2.out‚ 0.0390625 cpu‚25.283170352351444 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚47848.90478515625 disk.dm-1.out‚ 476.68359375ŽðÇÛ™)‚(:… ðôäÔ‹öÆx gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.508' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7016 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FÚ;Bdš)‚(:Ù ðôäÔžÿâz disk.rbd2.out‚ 303.546875$ memory_percent‚97.95217514038086 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.rbd0.in‚0! proc.memory.rssMB‚ 2862.765625 network.sent‚ 2.7014368e+07 disk.drbd1777.out‚0 disk.sda.in‚ 7.3671875 disk.sda.out‚ 477.0390625 disk.rbd2.in‚0 network.recv‚ 3.632592e+06( disk.nvme0c0n1.out‚47948.00634765625 disk.rbd3.in‚0& disk.nvme0n1.out‚47948.00634765625 disk.rbd0.out‚0) proc.memory.percent‚17.47293472290039 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 5.27587890625 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125& proc.memory.availableMB‚ 335.515625) disk./.usagePercent‚73.73737698530772% disk./.usageGB‚1098.7402038574219 disk.rbd4.out‚0 disk.sda3.out‚477 disk.dm-1.in‚ 7.31640625 disk.drbd1777.in‚ 1.03515625 disk.rbd1.in‚0 cpu‚25.282808882961653 disk.sda1.out‚0 disk.sda3.in‚ 7.31640625# disk.dm-0.out‚47948.00634765625 disk.sda2.in‚0 disk.dm-1.out‚477iß4›)/ _wandbruntime‚10341  _runtime‚10341àH¿Êœ)Ä ' lr-AdamW/pg1‚3.1919643056661296e-05 ' lr-AdamW/pg2‚3.1919643056661296e-05 trainer/global_step‚10699 $ _timestamp‚1.788426871000791e+09 _runtime‚10341.750738989 _step‚876¢IuÏ)É trainer/global_step‚10699 $ _timestamp‚1.788426871000791e+09 _runtime‚10341.750738989 _step‚876 ' lr-AdamW/pg1‚3.1919643056661296e-05 ' lr-AdamW/pg2‚3.1919643056661296e-05ì4+5‹4ž)/ _wandbruntime‚10341  _runtime‚10341%Lð » Ÿ)ëO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 278313155 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 197133199 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 _runtime‚10341.751695591 _step‚877 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚474555691370496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20208268801024 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400925005960192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71991862 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443271419768832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423408024580096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187318890 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚598 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115327626 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71991264 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19863641395200 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884268710015554e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 475420682 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19805992676572 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19805769027424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 247849984 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20208020951040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423408024580096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19863413191680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 278313155 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 197133197 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19863413191680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 475446354 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423408006577152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚10699 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 271790080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 197133199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443271419768832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400924988920832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚474555419580416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42340804%´606577152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19863641395200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443271665975296 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 197133197 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420730757948256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443271665975296 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19805992676572 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19805769027424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 247849984 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20208020951040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423408024580096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19863413191680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 278313155 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 197133197 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19863413191680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 475446354 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423408006577152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚10699 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 271790080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 197133199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443271419768832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400924988920832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚474555419580416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423408006577152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19863641395200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443271665975296 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 197133197 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420730757948256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443271665975296 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 _runtime‚10341.751695591 _step‚877 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚474555691370496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20208268801024 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400925005960192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71991862 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443271419768832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423408024580096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187318890 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚598퇩W«4¡)/ _wandbruntime‚10342  _runtime‚10342E ÿëû&¢)õM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 _step‚878 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 475463952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71997801 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 trainer/global_step‚10699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 475489650 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443311841627136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 197150916 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 % _timestamp‚1.7884268718659785e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 475489650 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187335782 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚454388568325632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19865479102976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚474598467392512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 197150916 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚443311841627136 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423446362524160 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 278338732 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚400961308418048 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 278313036 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19865250886144 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115337981 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚400961291378688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚443311595407360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420768892805712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20209899066880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187336416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71998433 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 197150918 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 197150918 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 278313036 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚454388592265728 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073G‡Øs'£)úM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19865250886144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115337983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 475463952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71997801 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 trainer/global_step‚10699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 475489650 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443311841627136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 197150916 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 % _timestamp‚1.7884268718659785e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 475489650 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187335782 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚454388568325632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19865479102976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚474598467392512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 197150916 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚443311841627136 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423446362524160 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 278338732 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚400961308418048 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 278313036 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19865250886144 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115337981 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚400961291378688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚443311595407360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420768892805712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20209899066880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187336416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71998433 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 197150918 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 197150918 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 278313036 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚454388592265728 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309050880 _step‚878 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445280462315520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115845378 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 279568387 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚476707450662912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788426916375358e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 279594057 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 477613206 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚422637514412264 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 198019149 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476707743424512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 trainer/global_step‚10749 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425324903780352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚456405907382272 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19955540532224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445280216109056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚425324903780352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72317051 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚402740128852992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚445280216109056 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 198019147 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445280462315520 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 198019147 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 279594057 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425324921783296 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20301836042240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚456405883442176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19897402598632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 188162429 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115845376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445280216109056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚425324903780352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72317051 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚402740128852992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚445280216109056 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 198019147 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445280462315520 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 198019147 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 279594057 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425324921783296 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20301836042240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚456405883442176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19897402598632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 188162429 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115845376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445280462315520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115845378 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 279568387 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚476707450662912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788426916375358e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 279594057 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 477613206 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚422637514412264 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 198019149ñ½À±E4³)/ _wandbruntime‚10387  _runtime‚10387å·ã´)õM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20303725671936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚10387.978368535 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115855731 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 198036868 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 477656502 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 279593938 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198036868 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425363259727360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198036866 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 279619634 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚476750505807872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>Devic?BTeStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚445320637967360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚456447077078528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 198036866 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚476750802750464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚445320391747584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72322568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72323207 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 279593938 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚882 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19899458647588 % _timestamp‚1.7884269172276733e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 279619634 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚402776414271488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19957378240000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20303452669440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 477630804 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 477656502 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188178299 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚422675889958436 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425363259727360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445320637967360¨Ms¡'µ)úM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 477656502 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 279593938 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198036868 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425363259727360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198036866 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 279619634 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚476750505807872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚445320637967360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚456447077078528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 198036866 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚476750802750464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚445320391747584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72322568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72323207 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 279593938 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚882 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19899458647588 % _timestamp‚1.7884269172276733e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 279619634 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚402776414271488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19957378240000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20303452669440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 477630804 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 477656502 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188178299 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚422675889958436 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425363259727360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445320637967360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20303725671936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚10387.978368535 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115855731 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 198036868 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0òàñ–4¶)/ _wandbruntime‚10387  _runtime‚10387Ž§]i[·)Õ .loss/FBt1/detection_loss‚2.128906063735485 % _timestamp‚1.7884269172296677e+09 /loss/FRt1/assignment_loss‚2.860732834253992 1loss/SRqqt2/detection_loss‚0.6465787566343292 _runtime‚10387.979482273 epoch‚8 /loss/FRt2/detection_loss‚1.7033841602859043 1loss/SRqqt1/detection_loss‚2.5228178470562668 %loss/total_loss‚38.88371591743202 _step‚883 .loss/FBt3/detection_loss‚2.322221902283755 /loss/FBt4/detection_loss‚2.4666509893205433 trainer/global_step‚10749 .loss/FRt1/detection_loss‚2.767185622294034 0loss/FRt2/assignment_loss‚0.8143404835746402 /loss/FRt3/assignment_loss‚2.583521524141001 0loss/FRt4/assignment_loss‚3.3626630890126132 2loss/SRqqt4/assignment_loss‚1.3440195215357975 0loss/FBt1/assignment_loss‚0.7984215434225916 /loss/FRt3/detection_loss‚2.3597484002279683 .loss/FRt4/detection_loss‚2.185480870848587 /loss/FBt2/detection_loss‚0.6906525504838844 1loss/FBt4/assignment_loss‚0.10966717121967424 2loss/SRqqt2/assignment_loss‚0.7524295767471499 2loss/SRqqt1/assignment_loss‚1.5485634191009479 1loss/FBt3/assignment_loss‚0.16278208772101524 1loss/SRqqt3/assignment_loss‚1.107646137682928 1loss/FBt2/assignment_loss‚0.11363196509109509 /loss/SRqqt3/detection_loss‚1.58966931645517 1loss/SRqqt4/detection_loss‚1.9420000843026421F¡Ð¸`¸)Ú % _timestamp‚1.7884269172296677e+09 /loss/FRt1/assignment_loss‚2.860732834253992 1loss/SRqqt2/detection_loss‚0.6465787566343292 _runtime‚10387.979482273 epoch‚8 /loss/FRt2/detection_loss‚1.7033841602859043 1loss/SRqqt1/detection_loss‚2.5228178470562668 %loss/total_loss‚38.88371591743202 _step‚883 .loss/FBt3/detection_loss‚2.322221902283755 /loss/FBt4/detection_loss‚2.4666509893205433 trainer/global_step‚10749 .loss/FRt1/detection_loss‚2.767185622294034 0loss/FRt2/assignment_loss‚0.8143404835746402 /loss/FRt3/assignment_loss‚2.583521524141001 0loss/FRt4/assignment_loss‚3.3626630890126132 2loss/SRqqt4/assignment_loss‚1.3440195215357975 0loss/FBt1/assignment_loss‚0.7984215434225916 /loss/FRt3/detection_loss‚2.3597484002279683 .loss/FRt4/detection_loss‚2.185480870848587 /loss/FBt2/detection_loss‚0.6906525504838844 1loss/FBt4/assignment_loss‚0.10966717121967424 2loss/SRqqt2/assignment_loss‚0.7524295767471499 2loss/SRqqt1/assignment_loss‚1.5485634191009479 1loss/FBt3/assignment_loss‚0.16278208772101524 1loss/SRqqt3/assignment_loss‚1.107646137682928 1loss/FBt2/assignment_loss‚0.11363196509109509 /loss/SRqqt3/detection_loss‚1.58966931645517 1loss/SRqqt4/detection_loss‚1.9420000843026421 .loss/FBt1/detection_loss‚2.128906063735485ó¶{0¹)‚(:… ¬õäÔɪy gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.872' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öF+hº)‚(:Ý ¬õäÔøØz% disk./.usageGB‚1098.7442016601562 disk.dm-0.out‚ 48247.5390625 disk.sda2.in‚0 disk.dm-0.in‚5.3125 disk.rbd2.in‚0 disk.rbd4.in‚0$ disk.nvme0c0n1.out‚ 48247.5390625 disk.sda3.in‚ 7.31640625 disk.nvme0n1.in‚5.3125$ memory_percent‚97.95329570770264 cpu‚25.281468929252533 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda3.out‚ 479.484375 disk.rbd1.out‚ 5.27587890625 disk.rbd2.out‚ 308.484375 disk.drbd1777.in‚ 1.03515625 disk.rbd3.out‚0" disk.nvme0n1.out‚ 48247.5390625) disk./.usagePercent‚73.73764528120358 disk.dm-1.out‚ 479.484375 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.sda.in‚ 7.3671875! proc.memory.rssMB‚ 2862.765625 network.sent‚ 2.7139828e+07 network.recv‚ 3.651424e+06 disk.sda2.out‚ 0.0390625 disk.sda1.out‚0) proc.memory.percent‚17.47293472290039 disk.dm-1.in‚ 7.31640625 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda.out‚ 479.5234375( proc.memory.availableMB‚ 335.33203125g¾»)‚(:‚ »õäÔ€ìµx gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚90.32' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.064 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥zË—s¼)‚(:è »õäÔîïƒ{ disk.rbd0.out‚0 network.recv‚ 3.658773e+06 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 480.11328125 disk.sda3.out‚ 480.07421875 disk.sda2.in‚0 disk.sda.in‚ 7.3671875) proc.memory.percent‚17.47293472290039 disk.rbd4.out‚0! proc.memory.rssMB‚ 2862.765625! disk.dm-0.out‚48251.658203125 disk.dm-1.out‚ 480.07421875 disk.rbd2.out‚ 308.51171875 disk.rbd3.in‚0$ disk.nvme0n1.out‚48251.658203125( proc.memory.availableMB‚ 335.37890625 disk.dm-0.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda3.in‚ 7.31640625$ memory_percent‚97.95300960540771 disk.dm-1.in‚ 7.31640625 disk.nvme0n1.in‚5.3125 cpu‚25.28113452951601 proc.cpu.threads‚17 network.sent‚ 2.7248819e+07) disk./.usagePercent‚73.73764220911319$ disk./.usageGB‚1098.744155883789 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0& disk.nvme0c0n1.out‚48251.658203125 disk.rbd1.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 5.27587890625Žp/£›½)‚(: ÊõäÔµŒ¼x gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.663' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.732599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08óøXw¾)‚(:ì ÊõäÔ»¶™}& disk.nvme0c0n1.out‚48350.830078125 disk.sda.in‚ 7.3671875 disk.sda.out‚ 480.79296875 disk.sda3.in‚ 7.31640625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd3.in‚0$ disk.nvme0n1.out‚48350.830078125 disk.sda2.in‚0 disk.dm-1.out‚ 480.75390625( proc.memory.availableMB‚ 335.12890625) proc.memory.percent‚17.47293472290039 disk.dm-0.in‚5.3125 disk.dm-1.in‚ 7.31640625 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 5.27587890625 disk.drbd1777.out‚0 proc.cpu.threads‚17 network.recv‚ 3.662921e+06) disk./.usagePercent‚73.73764092907551% disk./.usageGB‚1098.7441368103027 disk.nvme0n1.in‚5.3125$ memory_percent‚97.95453548431396 network.sent‚ 2.725488e+07 disk.sda3.out‚ 480.75390625 disk.rbd1.in‚0! disk.dm-0.out‚48350.830078125! proc.memory.rssMB‚ 2862.765625 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125! disk.rbd2.out‚310.23876953125 disk.rbd0.out‚0 cpu‚25.280791891120344¤.šÑ4¿)/ _wandbruntime‚10432  _runtime‚10432´Ì‚ÊÉÀ)à trainer/global_step‚10799 % _timestamp‚1.7884269618235967e+09 _runtime‚10432.573532983 _step‚884 & lr-AdamW/pg1‚2.754656322484727e-05 & lr-AdamW/pg2‚2.754656322484727e-058²ZÎÁ)È trainer/global_step‚10799 % _timestamp‚1.7884269618235967e+09 _runtime‚10432.573532983 _step‚884 & lr-AdamW/pg1‚2.754656322484727e-05 & lr-AdamW/pg2‚2.754656322484727e-05ôÅ™3 4Â)/ _wandbruntime‚10432  _runtime‚10432' •r€%Ã)ëO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424544511564780 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚447289012449280 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 198905097 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 280874959 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚427241800983552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72643082 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚427241818986496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚478859838730752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚424544270876272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚447289258655744 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 189006838 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚10799 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 280849289 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116363128 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚427241818986496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚447289012449280 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20047211465728 % _timestamp‚1.7884269618243096e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 189006208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458464392195072 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19989036169840 _step‚885 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 198905099 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19989259818988 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 479754386 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 280849289 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20395446535680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20395175617024 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 189006208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458464392195072 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19989036169840 _step‚885 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 198905099 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19989259818988 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 479754386 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 280849289 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20395446535680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20395175617024 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 270918656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚447289012449280 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 198905097 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 280874959 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚427241800983552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72643082 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚427241818986496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚478859838730752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚424544270876272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚447289258655744 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 189006838 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚10799 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 280849289 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116363128 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚427241818986496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚447289012449280 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20047211465728 % _timestamp‚1.7884269618243096e+09õ÷úÅø4Å)/ _wandbruntime‚10433  _runtime‚10433Ÿª¡fú&Æ)ôM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427280138927616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚478902599737856 % _timestamp‚1.7884269626877265e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚646 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19990868569440 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447329188087808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚427280138927616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20049277377024 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20049049160192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447329434307584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20397334789120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458505561891328 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 280900536 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427280156930560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72649782 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424582646422444 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚404591554203648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 479797656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189023263 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116373481 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 479823354 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198922816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚458505537951232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 280874840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72650428 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 479823354 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 _step‚886 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚478902896680448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716qmÿ&Ç)ùM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198922818 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 479823354 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198922816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚458505537951232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 280874840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72650428 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 479823354 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 _step‚886 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚478902896680448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427280138927616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚478902599737856 % _timestamp‚1.7884269626877265e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚646 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19990868569440 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447329188087808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚427280138927616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20049277377024 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20049049160192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447329434307584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20397334789120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458505561891328 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 280900536 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427280156930560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72649782 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424582646422444 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚404591554203648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 479797656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189023263 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116373481 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚429158716189696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72963500 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚449297808789504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 199791047 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚429158716189696 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406370374638592 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚449297808789504 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20139110602752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚426451268028788 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20139338806272 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚481011834597376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20139110602752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 _runtime‚10477.939023693 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 282130191 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚666 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚429158698186752 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 199791047 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561‹8È‚ö'Ö)ðO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20139110602752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚426451268028788 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20139338806272 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚481011834597376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20139110602752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 _runtime‚10477.939023693 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 282130191 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚666 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚429158698186752 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 199791047 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 282130191 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116880878 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚460522877007872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚426451027340280 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚406370357599232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚449298054995968 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 116880876 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 199791049 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72962834 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚481011533447168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309050880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚449298054995968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429158698186752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884270071889522e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚429158716189696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72963500 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚449297808789504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 199791047 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚429158716189696 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406370374638592 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚449297808789504 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 481990206 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20082502140648 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 199808768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 199808766 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚481054575161856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚429197036130816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20490552397824 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 282155742 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚406406660057088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20141176514048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 282181438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚460564046704128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚449337984428032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚460564022764032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 282155742 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72970206 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20082725790004 _runtime‚10478.808574553 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚890 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189860786 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 189861439 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 % _timestamp‚1.7884270080581362e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 282181438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚429197054133760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20141176514048 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20140948297216 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚481054867910144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚406406677096448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚449338230647808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚426489162197736ù\(&'Ù)úM F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚429197054133760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20141176514048 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20140948297216 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚481054867910144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚406406677096448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚449338230647808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚426489162197736 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 481964508 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚449338230647808 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 481990206 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20082502140648 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 199808768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 199808766 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚481054575161856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚429197036130816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20490552397824 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 282155742 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚406406660057088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20141176514048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 282181438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚460564046704128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚449337984428032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚460564022764032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 282155742 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72970206 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20082725790004 _runtime‚10478.808574553 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚890 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189860786 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 268808192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 189861439 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 % _timestamp‚1.7884270080581362e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 282181438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2úò¬«à4Ú)/ _wandbruntime‚10478  _runtime‚10478“ö¿_Û)Ù /loss/FRt3/assignment_loss‚2.446682340059525 2loss/SRqqt3/assignment_loss‚1.5173612111671404 -loss/FBt2/detection_loss‚0.62180550775019 &loss/total_loss‚40.081848979452396 trainer/global_step‚10849 1loss/FBt3/assignment_loss‚0.26471299323020503 /loss/FRt2/detection_loss‚1.3244191384782977 2loss/SRqqt1/assignment_loss‚1.4377240467195709 2loss/SRqqt4/assignment_loss‚1.3181140607268345 0loss/SRqqt4/detection_loss‚2.069398136764038 .loss/FBt1/detection_loss‚3.039014783772555 /loss/FBt3/detection_loss‚2.1717801213264467 _step‚891 0loss/FRt1/assignment_loss‚3.1588947527548847 /loss/FRt1/detection_loss‚2.5699338106548084 /loss/FRt3/detection_loss‚2.5105930994718504 /loss/FBt4/detection_loss‚2.2696838312678866 /loss/FRt4/assignment_loss‚3.090785887895846 2loss/SRqqt2/assignment_loss‚0.6121338800636723 1loss/FBt1/assignment_loss‚0.10312215282233823 /loss/FRt4/detection_loss‚2.1619907782102503 % _timestamp‚1.7884270080598361e+09 0loss/FRt2/assignment_loss‚1.2022243294061399 1loss/FBt4/assignment_loss‚0.20874419090816648 1loss/FBt2/assignment_loss‚0.07793015444018024 1loss/SRqqt2/detection_loss‚0.9095946537769016 _runtime‚10478.809645881 0loss/SRqqt1/detection_loss‚2.922869530816873 1loss/SRqqt3/detection_loss‚2.0723355869677933 epoch‚8áÎndÜ)Þ 2loss/SRqqt3/assignment_loss‚1.5173612111671404 -loss/FBt2/detection_loss‚0.62180550775019 &loss/total_loss‚40.081848979452396 trainer/global_step‚10849 1loss/FBt3/assignment_loss‚0.26471299323020503 /loss/FRt2/detection_loss‚1.3244191384782977 2loss/SRqqt1/assignment_loss‚1.4377240467195709 2loss/SRqqt4/assignment_loss‚1.3181140607268345 0loss/SRqqt4/detection_loss‚2.069398136764038 .loss/FBt1/detection_loss‚3.039014783772555 /loss/FBt3/detection_loss‚2.1717801213264467 _step‚891 0loss/FRt1/assignment_loss‚3.1588947527548847 /loss/FRt1/detection_loss‚2.5699338106548084 /loss/FRt3/detection_loss‚2.5105930994718504 /loss/FBt4/detection_loss‚2.2696838312678866 /loss/FRt4/assignment_loss‚3.090785887895846 2loss/SRqqt2/assignment_loss‚0.6121338800636723 1loss/FBt1/assignment_loss‚0.10312215282233823 /loss/FRt4/detection_loss‚2.1619907782102503 % _timestamp‚1.7884270080598361e+09 0loss/FRt2/assignment_loss‚1.2022243294061399 1loss/FBt4/assignment_loss‚0.20874419090816648 1loss/FBt2/assignment_loss‚0.07793015444018024 1loss/SRqqt2/detection_loss‚0.9095946537769016 _runtime‚10478.809645881 0loss/SRqqt1/detection_loss‚2.922869530816873 1loss/SRqqt3/detection_loss‚2.0723355869677933 epoch‚8 /loss/FRt3/assignment_loss‚2.446682340059525ûnpp_›Ý)‚(: †öäÔ¼Á©x gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.775' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.755000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09Bc!yÞ)‚(:î †öäÔ¢ËËz disk.rbd4.out‚0 disk.sda.out‚ 483.1484375 cpu‚25.279440014488078 disk.rbd1.in‚0# disk.dm-0.out‚48649.42431640625 disk.sda2.out‚ 0.0390625 network.sent‚ 2.7381359e+07! disk.rbd2.out‚313.61474609375 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚48649.42431640625) proc.memory.percent‚17.47293472290039 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0! proc.memory.rssMB‚ 2862.765625 proc.cpu.threads‚17 disk.dm-1.in‚ 7.31640625 disk.nvme0n1.in‚5.3125 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.3671875 network.recv‚ 3.682971e+06& disk.nvme0n1.out‚48649.42431640625 disk.sda2.in‚0 disk.dm-1.out‚ 483.109375 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0) disk./.usagePercent‚73.73764092907551 disk.rbd1.out‚ 5.27587890625 disk.dm-0.in‚5.3125 disk.drbd1777.out‚0 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 483.109375$ memory_percent‚97.95491695404053( proc.memory.availableMB‚ 335.06640625% disk./.usageGB‚1098.7441368103027™"ß)‚(:„ •öäÔ ’žx gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.403' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6806 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`àKZ}à)‚(:ò •öäÔ©€Ð{ disk.dm-1.out‚ 483.97265625 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.rbd2.in‚0 network.sent‚ 2.7490592e+07 disk.rbd3.in‚0 disk.sda3.out‚ 483.97265625 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0! proc.memory.rssMB‚ 2862.765625) disk./.usagePercent‚73.73764092907551 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 disk.dm-1.in‚ 7.31640625! disk.rbd2.out‚315.14990234375 cpu‚25.279107946679368 proc.cpu.threads‚17 network.recv‚ 3.69069e+06 disk.rbd1.out‚ 5.27587890625( proc.memory.availableMB‚ 335.13671875% disk./.usageGB‚1098.7441368103027 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda3.in‚ 7.31640625# disk.dm-0.out‚48748.58251953125 disk.rbd0.out‚0$ memory_percent‚97.95448780059814 disk.sda.in‚ 7.3671875 disk.sda2.in‚0 disk.rbd0.in‚0) proc.memory.percent‚17.47293472290039( disk.nvme0c0n1.out‚48748.58251953125 disk.sda.out‚ 484.01171875 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚48748.58251953125¶¿Àá)‚(:… ¤öäÔú÷¤x gpu.0.gpu‚46.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚127.583' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓÒNvâ)‚(:ë ¤öäÔ•Ë÷z% disk./.usageGB‚1098.7441368103027 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 484.40625 disk.rbd4.out‚0 disk.sda.in‚ 7.3671875 disk.sda.out‚ 484.4453125) proc.memory.percent‚17.47293472290039 cpu‚25.28208184021575 disk.dm-1.in‚ 7.31640625 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 484.40625) disk./.usagePercent‚73.73764092907551 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.sda3.in‚ 7.31640625# disk.dm-0.out‚48748.87939453125( proc.memory.availableMB‚ 335.13671875 disk.rbd1.out‚ 5.27587890625 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚48748.87939453125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0! proc.memory.rssMB‚ 2862.765625 network.sent‚ 2.7495884e+07 network.recv‚ 3.694073e+06 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125! disk.rbd2.out‚315.29052734375& disk.nvme0n1.out‚48748.87939453125 disk.rbd4.in‚0$ memory_percent‚97.95448780059814 proc.cpu.threads‚17µ9ýã)‚(:„ ³öäÔ„ˆ³x gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.181' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žCñ ~ä)‚(:ó ³öäÔz disk.rbd4.out‚0 network.sent‚ 2.7501914e+07 disk.dm-1.in‚ 7.31640625 disk.dm-1.out‚ 484.97265625 disk.rbd4.in‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.3671875 disk.sda3.in‚ 7.31640625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 5.27587890625 disk.sda2.out‚ 0.0390625) proc.memory.percent‚17.47293472290039) disk./.usagePercent‚73.73770595498921 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.sda.out‚ 485.01171875# disk.dm-0.out‚48848.17138671875 disk.sda2.in‚0( disk.nvme0c0n1.out‚48848.17138671875 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚48848.17138671875 network.recv‚ 3.698299e+06% disk./.usageGB‚1098.7451057434082 disk.rbd0.in‚0 disk.sda3.out‚ 484.97265625$ memory_percent‚97.95510768890381! disk.rbd2.out‚316.81005859375 disk.dm-0.in‚5.3125 cpu‚25.288647514467264 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 335.03515625! proc.memory.rssMB‚ 2862.765625gW¥šå)‚(: ÂöäÔ¹¹›x gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.563' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.712600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äôetæ)‚(:é ÂöäÔì¾·z disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 5.27587890625 disk.dm-1.out‚ 485.3046875$ memory_percent‚97.95355796813965 disk.dm-1.in‚ 7.31640625 cpu‚25.294645274794803 disk.nvme0c0n1.in‚5.3125 disk.nvme0n1.in‚5.3125 disk.sda.out‚ 485.34375% disk.nvme0n1.out‚48947.4404296875 disk.dm-0.in‚5.3125" disk.dm-0.out‚48947.4404296875 disk.sda2.out‚ 0.0390625 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚316.91552734375' proc.memory.availableMB‚ 335.2890625 network.recv‚ 3.702341e+06 disk.rbd4.in‚0' disk.nvme0c0n1.out‚48947.4404296875 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.3671875! proc.memory.rssMB‚ 2862.765625) proc.memory.percent‚17.47293472290039 network.sent‚ 2.7507887e+07$ disk./.usageGB‚1098.745204925537 disk.rbd0.in‚0 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 485.3046875 disk.drbd1777.in‚ 1.03515625 disk.rbd3.in‚0) disk./.usagePercent‚73.73771261118509 disk.rbd0.out‚0 disk.rbd4.out‚0ž7ë4ç)/ _wandbruntime‚10558  _runtime‚10558#Ñ`•Âè)ÑT bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚431666429097984 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚408744930763776 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 537294673 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚299 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 6893338624 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 549453824 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚463183641889792 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 164704768 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚561 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20201 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚451883842038272 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 283359557 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 191020590 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚408744947803136 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚20217241229312 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚561 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 189713920 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚20217412940288 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31397 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20201 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 117651474 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 484469679 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 549453824 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20203 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚20217241229312 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 184860760 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 484449476 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 201089921 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚483751709104640 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337968128 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6039636992 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 6893338624 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚262 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚431666429097984 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚299 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚374 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚451883652324352 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6739344896 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚483751544399872 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 140764672 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_validation_batch_s³ ’• tart/requested_bytes.all.peak‚ 6522528073 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚20158552014392 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 201089921 trainer/global_step‚10880 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚431666411095040 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _runtime‚10558.93696615 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 283379758 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 171710976 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 201089919 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1243 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚262 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚561 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 6893338624 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6343884800 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚299 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚451883652324352 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 283379758 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 191020964 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1261 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚73369118 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚20568067214848 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 283359557 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6739344896 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 541892608 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31168 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚73369490 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚451883842038272v$Q\*é)ÖT % _timestamp‚1.7884270881866596e+09 _step‚892 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 171710976 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚20567926450176 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ^QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚428903482778168 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚20158719835792 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20203 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6252512256 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31397 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20201 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 117651474 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 484469679 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 549453824 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20203 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚20217241229312 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 184860760 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 484449476 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 201089921 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚483751709104640 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337968128 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6039636992 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 6893338624 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚262 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚431666429097984 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚299 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚374 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚451883652324352 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6739344896 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚483751544399872 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 140764672 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6522528073 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚20158552014392 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 201089921 trainer/global_step‚10880 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚431666411095040 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _runtime‚10558.93696615 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 283379758 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 171710976 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 201089919 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1243 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚262 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚561 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 6893338624 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6343884800 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚299 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚451883652324352 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 283379758 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 191020964 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1261 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚73369118 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚20568067214848 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 283359557 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6739344896 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 541892608 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31168 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚73369490 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚451883842038272 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚431666429097984 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚408744930763776 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 537294673 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚299 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 6893338624 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 549453824 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚463183641889792 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 164704768 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚561 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20201 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚451883842038272 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 283359557 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 191020590 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚408744947803136 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚20217241229312 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚561 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 189713920 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚20217412940288 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 283367833 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚483779516891648 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 549453824 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚408769700752384 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 167821400 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 6893338624 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 171710976 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 trainer/global_step‚10880 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 189713920 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 541892608 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 117660438 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 283367833 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 283388034 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚428928626904312 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 484472688 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚20217634024448 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚23 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 164704768 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚561 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20203 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 191034087 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 541892608 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 140764672 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 201104857 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚373 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚451910375623168 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31397 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1261 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1243 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚20158943191288 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚431692551884800 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3428 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 184860760 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚408769683713024 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚299 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 201104855 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚20217634024448 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31168 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 537294673 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20203 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚375 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚73373276 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 6893338624 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 283367833 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚483779516891648 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 549453824 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚408769700752384 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 167821400 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 6893338624 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 171710976 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 trainer/global_step‚10880 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 189713920 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 541892608 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 117660438 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 283367833 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 283388034 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚428928626904312 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 484472688 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚20217634024448 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚23 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 164704768 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚561 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20203 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 191034087 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 549453824 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 541892608 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 140764672 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 201104857 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚373 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚451910375623168 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31397 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1261 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1243 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚20158943191288 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚431692551884800 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3428 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 184860760 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚408769683713024 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚299 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 201104855 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚20217634024448 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31168 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6522528073 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 537294673 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20203 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚375 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚73373276 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 Q×RÁBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 6893338624 B&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2476319819688797 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10160228610038757 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.39367812871932983 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13759908080101013 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14133626222610474 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4732142686843872 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.37654322385787964 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 % _timestamp‚1.7884271031515515e+09 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4912163019180298 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4393274188041687 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818182468414307 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04885800555348396 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25475695729255676 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 +particle/specificity‚0.9513136744499207 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.522149384021759 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.023998146876692772 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4360036551952362 particle/accuracy_2_of_2‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5150089859962463 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.44487011432647705 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.06578947603702545 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5234410762786865 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43721556663513184 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20113424956798553 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1795995682477951 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.38477852940559387 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3670700788497925 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 particle/accuracy_3_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9798515439033508 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9950113296508789 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.992901623249054 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.08575330674648285 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20488505065441132 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.47840163111686707 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47614943981170654 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 jet/accuracy_2_of_7‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4786232113838196 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.37037035822868347 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4696304500102997 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9883551597595215 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4912656247615814 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5764996409416199 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1736111044883728 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9907070994377136 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17779946327209473 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03951149433851242 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.42168867588043213 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8192809224128723 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38424062728881836 6 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.101259745657444 particle/accuracy_1_of_2‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.13461539149284363 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.39666667580604553 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1260705590248108 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4547618627548218 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1489478051662445 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3973179757595062 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.042635660618543625 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.44706860184669495 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.018881207332015038 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4969613254070282 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24116161465644836 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.41674211621284485 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0037772480864077806 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984307885169983 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11960510909557343 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3392069637775421 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25306373834609985 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6711050271987915 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4726240634918213 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.004232804290950298 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 trainer/global_step‚10880 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5465278029441833 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14707310497760773 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_4_of_5‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4531366527080536 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5209434628486633 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.45502912998199463 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.44736841320991516 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ) Purity/3FRt1SRqqt1FBt/FRt_purity‚0.25 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 _step‚894 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424243688583374 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9889103770256042 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.494047611951828 *jet/accuracy_1_of_2‚0.7317314147949219 +jet/accuracy_2_of_3‚0.22928830981254578 7validation_average_jet_accuracy‚0.41841596364974976 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_1_of_4‚0.9116663932800293 particle/accuracy_3_of_3‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20397335290908813 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.398858904838562 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 (loss/val_total_loss‚41.1369888663718 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23389144241809845 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9798850417137146 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08383964747190475 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3958113193511963 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07311297208070755 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4128788113594055 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481483578681946 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9902384877204895 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9828309416770935 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_4_of_7‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18753200769424438 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.023479321971535683 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5260134935379028 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 particle/accuracy_5_of_6‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02014083042740822 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9134280681610107 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1727062463760376 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.40688034892082214 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_5‚0.058218128979206085 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 jet/accuracy_5_of_5‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14845307171344757 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 'particle/f_score‚0.4050632417201996 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 epoch‚8 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14716582000255585 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 particle/accuracy_5_of_5‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5266667008399963 particle/accuracy_6_of_6‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975067377090454 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.007823129184544086 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019773701205849648 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.005017921328544617 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47299903631210327 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39586055278778076 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6440190076828003 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 particle/accuracy_5_of_7‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.849731981754303 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13546743988990784 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4624345600605011 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2888888716697693 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13939712941646576 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 3"Purity/0FRt*SRqqt*FBt/event_purity‚ 0.4541015625 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2727891802787781 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34583333134651184 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908720850944519 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13018706440925598 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49344250559806824 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14212815463542938 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2065664380788803 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.015476190485060215 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3727697432041168 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4445450007915497 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +particle/sensitivity‚0.3209478259086609 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.40214425325393677 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5124912858009338 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9870827794075012 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3626384437084198 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36808037757873535 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.44036412239074707 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9922763109207153 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 +jet/accuracy_4_of_6‚0.01923076994717121 particle/accuracy_7_of_7‚1 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006430658511817455 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23423424363136292 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10056287795305252 _runtime‚10573.903584906 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4796609580516815 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 +jet/accuracy_2_of_4‚0.34869250655174255 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_1_of_6‚1 jet/accuracy_5_of_7‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5582097172737122 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16638624668121338 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/0°æsË!FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13600558042526245 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.058242522180080414 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.013315597549080849 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4166666567325592 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4048821032047272 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25462964177131653 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1041061133146286 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009906730614602566 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9930983781814575 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16316364705562592 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2460317611694336 particle/accuracy_2_of_4‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.04609929397702217 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2300972044467926 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3155519962310791 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4524086117744446 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5069215893745422 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.10000000149011612 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5047115087509155 particle/accuracy_2_of_3‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05850942060351372 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4855037033557892 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5434680581092834 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16316364705562592 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14927595853805542 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24694958329200745 +jet/accuracy_3_of_3‚0.01653757505118847 ,jet/accuracy_4_of_4‚0.004316743463277817 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011237087659537792 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_2_of_5‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.48085686564445496 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 jet/accuracy_6_of_6‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4531366527080536 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23742511868476868 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5508928894996643 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_4_of_6‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968628287315369 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.37496164441108704 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09555555135011673 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3923611342906952 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16728417575359344 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3753472566604614 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5298534631729126 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13813504576683044 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10536398738622665 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5134949088096619 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9910677671432495 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4124317765235901 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40000003576278687 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28920844197273254 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25684356689453125 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17329296469688416 jet/accuracy_5_of_6‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 particle/accuracy_4_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4622582495212555 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5501298308372498 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5018872022628784 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455MAiØ'^ñ)ݦ 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4524086117744446 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5069215893745422 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.10000000149011612 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5047115087509155 particle/accuracy_2_of_3‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05850942060351372 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4855037033557892 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5434680581092834 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16316364705562592 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14927595853805542 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24694958329200745 +jet/accuracy_3_of_3‚0.01653757505118847 ,jet/accuracy_4_of_4‚0.004316743463277817 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011237087659537792 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_2_of_5‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.48085686564445496 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 jet/accuracy_6_of_6‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4531366527080536 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23742511868476868 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5508928894996643 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_4_of_6‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968628287315369 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.37496164441108704 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09555555135011673 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3923611342906952 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16728417575359344 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3753472566604614 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5298534631729126 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13813504576683044 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10536398738622665 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5134949088096619 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9910677671432495 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4124317765235901 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40000003576278687 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28920844197273254 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25684356689453125 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17329296469688416 jet/accuracy_5_of_6‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 particle/accuracy_4_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4622582495212555 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5501298308372498 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5018872022628784 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333134651184 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4529799818992615 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9907435774803162 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.15552955865859985 +jet/accuracy_3_of_4‚0.05930744484066963 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9918023347854614 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1505747139453888 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 particle/accuracy_1_of_1‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.21666666865348816 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19873160123825073 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18020302057266235 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.21984517574310303 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19476422667503357 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11728395521640778 jet/accuracy_6_of_7‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5369011759757996 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3232135772705078 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 (particle/accuracy‚0.8528593182563782 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.00741239869967103 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_6‚0.47435900568962097 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0515873022377491 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5229006409645081 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1643274873495102 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5347222089767456 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.40840834379196167 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2476319819688797 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10160228610038757 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.39367812871932983 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13759908080101013 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14133626222610474 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4732142686843872 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.37654322385787964 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 % _timestamp‚1.7884271031515515e+09 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4912163019180298 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4393274188041687 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818182468414307 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04885800555348396 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25475695729255676 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 +particle/specificity‚0.9513136744499207 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.522149384021759 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.023998146876692772 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4360036551952362 particle/accuracy_2_of_2‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5150089859962463 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.44487011432647705 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.06578947603702545 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5234410762786865 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43721556663513184 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20113424956798553 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1795995682477951 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.38477852940559387 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3670700788497925 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 particle/accuracy_3_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9798515439033508 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9950113296508789 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.992901623249054 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.08575330674648285 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20488505065441132 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.47840163111686707 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47614943981170654 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 jet/accuracy_2_of_7‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4786232113838196 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.37037035822868347 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4696304500102997 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9883551597595215 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4912656247615814 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5764996409416199 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1736111044883728 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9907070994377136 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17779946327209473 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03951149433851242 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.42168867588043213 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8192809224128723 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38424062728881836 6 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.101259745657444 particle/accuracy_1_of_2‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.13461539149284363 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.39666667580604553 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1260705590248108 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4547618627548218 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1489478051662445 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3973179757595062 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.042635660618543625 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.44706860184669495 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.018881207332015038 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4969613254070282 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24116161465644836 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.41674211621284485 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0037772480864077806 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984307885169983 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11960510909557343 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3392069637775421 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25306373834609985 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6711050271987915 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4726240634918213 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.004232804290950298 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 trainer/global_step‚10880 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5465278029441833 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14707310497760773 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_4_of_5‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4531366527080536 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5209434628486633 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.45502912998199463 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.44736841320991516 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ) Purity/3FRt1SRqqt1FBt/FRt_purity‚0.25 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 _step‚894 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424243688583374 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9889103770256042 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.494047611951828 *jet/accuracy_1_of_2‚0.7317314147949219 +jet/accuracy_2_of_3‚0.22928830981254578 7validation_average_jet_accuracy‚0.41841596364974976 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_1_of_4‚0.9116663932800293 particle/accuracy_3_of_3‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20397335290908813 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.398858904838562 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 (loss/val_total_loss‚41.1369888663718 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23389144241809845 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9798850417137146 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08383964747190475 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3958113193511963 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07311297208070755 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4128788113594055 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481483578681946 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.990238487XN?=57204895 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9828309416770935 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_4_of_7‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18753200769424438 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.023479321971535683 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5260134935379028 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 particle/accuracy_5_of_6‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02014083042740822 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9134280681610107 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1727062463760376 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.40688034892082214 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_5‚0.058218128979206085 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 jet/accuracy_5_of_5‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14845307171344757 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 'particle/f_score‚0.4050632417201996 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 epoch‚8 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14716582000255585 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 particle/accuracy_5_of_5‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5266667008399963 particle/accuracy_6_of_6‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975067377090454 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.007823129184544086 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019773701205849648 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.005017921328544617 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47299903631210327 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39586055278778076 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6440190076828003 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 particle/accuracy_5_of_7‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.849731981754303 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13546743988990784 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4624345600605011 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2888888716697693 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13939712941646576 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 3"Purity/0FRt*SRqqt*FBt/event_purity‚ 0.4541015625 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2727891802787781 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34583333134651184 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908720850944519 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13018706440925598 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49344250559806824 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14212815463542938 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2065664380788803 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.015476190485060215 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3727697432041168 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4445450007915497 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +particle/sensitivity‚0.3209478259086609 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.40214425325393677 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5124912858009338 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9870827794075012 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3626384437084198 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36808037757873535 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.44036412239074707 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9922763109207153 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 +jet/accuracy_4_of_6‚0.01923076994717121 particle/accuracy_7_of_7‚1 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006430658511817455 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23423424363136292 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10056287795305252 _runtime‚10573.903584906 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4796609580516815 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 +jet/accuracy_2_of_4‚0.34869250655174255 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_1_of_6‚1 jet/accuracy_5_of_7‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5582097172737122 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16638624668121338 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13600558042526245 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.058242522180080414 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.013315597549080849 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4166666567325592 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4048821032047272 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25462964177131653 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1041061133146286 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009906730614602566 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9930983781814575 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16316364705562592 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2460317611694336 particle/accuracy_2_of_4‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.04609929397702217 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2300972044467926 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3155519962310791 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0þ¼O8wŽò)‚(:ƒ àöäԳ׎x gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.43' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x¼îE|ó)‚(:ñ àöäÔ­âúz! proc.memory.rssMB‚ 2860.796875# disk.dm-0.out‚49047.20556640625% disk./.usageGB‚1098.7462196350098 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.in‚ 7.3671875& disk.nvme0n1.out‚49047.20556640625 network.recv‚ 3.713546e+06 disk.drbd1777.in‚ 1.03515625 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 486.41015625 disk.sda3.in‚ 7.31640625 disk.dm-1.out‚ 486.37109375 proc.cpu.threads‚17) disk./.usagePercent‚73.73778070918921 disk.rbd4.in‚0( proc.memory.availableMB‚ 336.67578125 cpu‚25.306425037730712 disk.rbd2.out‚ 320.134765625 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.in‚0 disk.rbd1.out‚ 5.27587890625 disk.dm-0.in‚5.3125( disk.nvme0c0n1.out‚49047.20556640625 disk.rbd3.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.460918426513672 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 486.37109375 disk.nvme0n1.in‚5.3125 disk.dm-1.in‚ 7.31640625$ memory_percent‚97.94509410858154 network.sent‚ 2.762219e+07¢…ÿ ô)‚(:… ïöäÔÓØ•x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™„v2uõ)‚(:ê ïöäÔ¡™Ó{ disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0# memory_percent‚97.9457139968872$ disk.nvme0n1.out‚49146.509765625 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0 proc.cpu.threads‚17 disk.sda.in‚ 7.3671875 disk.rbd0.in‚0$ disk./.usageGB‚1098.746181488037 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚ 7.31640625* proc.memory.percent‚17.460918426513672! disk.dm-0.out‚49146.509765625! proc.memory.rssMB‚ 2860.796875 disk.sda3.in‚ 7.31640625 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.rbd1.out‚ 5.27587890625 disk.nvme0n1.in‚5.3125 disk.dm-1.out‚ 486.98046875 network.recv‚ 3.720982e+06) disk./.usagePercent‚73.73777814911386 disk.sda3.out‚ 486.98046875 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125( proc.memory.availableMB‚ 336.57421875 cpu‚25.306059351618305 network.sent‚ 2.7746341e+07 disk.sda.out‚ 487.01953125 disk.rbd2.out‚ 320.134765625& disk.nvme0c0n1.out‚49146.509765625¾¾±V4ö)/ _wandbruntime‚10590  _runtime‚10590hXxÆË÷)Å trainer/global_step‚10899 % _timestamp‚1.7884271195169716e+09 _runtime‚10590.266978186 _step‚895 ' lr-AdamW/pg1‚2.3484908534110592e-05 ' lr-AdamW/pg2‚2.3484908534110592e-05» “BÐø)Ê _step‚895 ' lr-AdamW/pg1‚2.3484908534110592e-05 ' lr-AdamW/pg2‚2.3484908534110592e-05 trainer/global_step‚10899 % _timestamp‚1.7884271195169716e+09 _runtime‚10590.266978186ÿYíت4ù)/ _wandbruntime‚10590  _runtime‚10590„âŹñ'ú)ëO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 201617967 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚432722483147776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884271195177383e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚432722465144832 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117963358 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚650 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 201617967 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚432722465144832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73540242 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 201617965 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20256001025024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 191504252 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117963360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 283960083 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 _step‚896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20256001025024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20197188213225 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚452978484172800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚652 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452978237966336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 283960083 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 485578050 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚409744916294656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚452978237966336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚432722483147776 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 191503600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚464308506308608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429941880858733 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20255772821504 _runtime‚10590.267821535\ûñ¾sû)ðO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚432722465144832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73540242 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 201617965 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20256001025024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 191504252 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117963360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 283960083 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 _step‚896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20256001025024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20197188213225 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301150208 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚452978484172800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚652 R>DeviceStatsM¿¥Îƒonitor.on_train_batch_start/active_bytes.all.freed‚452978237966336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 283960083 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 485578050 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚409744916294656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚452978237966336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚432722483147776 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 191503600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚464308506308608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429941880858733 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20255772821504 _runtime‚10590.267821535 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 201617967 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚432722483147776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 % _timestamp‚1.7884271195177383e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚432722465144832 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117963358 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚650 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 201617967 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0€ o"4ü)/ _wandbruntime‚10591  _runtime‚10591y‡RYû&ý)õM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409781235791872 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚453018413604864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚432760821091840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20257838732800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464349652064768 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚464349676004864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20609153099264 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 485621346 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20608875902464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 201635686 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20257610515968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚409781218752512 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20199020613033 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 485621346 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚453018659824640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 _step‚897 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73547190 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191520903 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 trainer/global_step‚10899 % _timestamp‚1.7884271203980176e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117973713 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73547835 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚484958829104128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚453018659824640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚432760803088896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20257838732800 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201635684 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20257610515968 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117973715 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 _runtime‚10591.148644285 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚432760821091840 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 201635686 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 283985660 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 283959964 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 283959964 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0¿îùg'þ)úM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 283985660 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 283959964 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 283959964 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409781235791872 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚453018413604864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚432760821091840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20257838732800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464349652064768 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚464349676004864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20609153099264 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 485621346 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20608875902464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 201635686 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20257610515968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚409781218752512 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20199020613033 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 485621346 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚453018659824640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 _step‚897 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73547190 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191520903 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 trainer/global_step‚10899 % _timestamp‚1.7884271203980176e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117973713 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73547835 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚484958829104128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚453018659824640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚432760803088896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20257838732800 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201635684 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20257610515968 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117973715 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 _runtime‚10591.148644285 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚432760821091840 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 201635686 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ø}“4ÿ)/ _wandbruntime‚10591  _runtime‚10591˜|S¹`€*Ú 2loss/SRqqt4/assignment_loss‚1.3725879239999423 /loss/FBt3/detection_loss‚2.7117618560791015 epoch‚9 trainer/global_step‚10899 2loss/SRqqt3/assignment_loss‚1.5905405921102673 1loss/FBt4/assignment_loss‚0.22689974733414905 1loss/SRqqt3/detection_loss‚1.8509979918599129 % _timestamp‚1.7884271203998716e+09 2loss/SRqqt2/assignment_loss‚0.7439038707962027 /loss/FRt2/detection_loss‚1.8083939821459354 _step‚898 1loss/FBt2/assignment_loss‚0.08566877678751798 .loss/FRt4/detection_loss‚2.137681204085997 0loss/SRqqt1/detection_loss‚2.480070303852965 /loss/FBt4/detection_loss‚3.3186058600743613 0loss/FRt2/assignment_loss‚1.4413218721747398 0loss/FBt2/detection_loss‚0.39690158285021065 %loss/total_loss‚40.61172046761522 0loss/FRt3/assignment_loss‚2.8800254415492623 1loss/SRqqt1/assignment_loss‚1.768086281128046 /loss/FRt1/detection_loss‚2.3556591303542604 /loss/FRt3/detection_loss‚2.4590036279084733 0loss/FRt4/assignment_loss‚2.7936304953363207 0loss/FBt1/assignment_loss‚0.1356479690409122 1loss/FBt3/assignment_loss‚0.45221105944365264 1loss/SRqqt4/detection_loss‚1.7958750825907503 .loss/FBt1/detection_loss‚2.099877352064306 _runtime‚10591.149697389 0loss/FRt1/assignment_loss‚2.7533715810886648 0loss/SRqqt2/detection_loss‚0.952996882959269Ù¹Õe*ß /loss/FBt4/detection_loss‚3.3186058600743613 0loss/FRt2/assignment_loss‚1.4413218721747398 0loss/FBt2/detection_loss‚0.39690158285021065 %loss/total_loss‚40.61172046761522 0loss/FRt3/assignment_loss‚2.8800254415492623 1loss/SRqqt1/assignment_loss‚1.768086281128046 /loss/FRt1/detection_loss‚2.3556591303542604 /loss/FRt3/detection_loss‚2.4590036279084733 0loss/FRt4/assignment_loss‚2.7936304953363207 0loss/FBt1/assignment_loss‚0.1356479690409122 1loss/FBt3/assignment_loss‚0.45221105944365264 1loss/SRqqt4/detection_loss‚1.7958750825907503 .loss/FBt1/detection_loss‚2.099877352064306 _runtime‚10591.149697389 0loss/FRt1/assignment_loss‚2.7533715810886648 0loss/SRqqt2/detection_loss‚0.952996882959269 2loss/SRqqt4/assignment_loss‚1.3725879239999423 /loss/FBt3/detection_loss‚2.7117618560791015 epoch‚9 trainer/global_step‚10899 2loss/SRqqt3/assignment_loss‚1.5905405921102673 1loss/FBt4/assignment_loss‚0.22689974733414905 1loss/SRqqt3/detection_loss‚1.8509979918599129 % _timestamp‚1.7884271203998716e+09 2loss/SRqqt2/assignment_loss‚0.7439038707962027 /loss/FRt2/detection_loss‚1.8083939821459354 _step‚898 1loss/FBt2/assignment_loss‚0.08566877678751798 .loss/FRt4/detection_loss‚2.137681204085997 0loss/SRqqt1/detection_loss‚2.480070303852965‚=CÄQ‚*‚(:… þöäÔÇØx gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñlÑ4uƒ*‚(:ê þöäÔîòÃ{ disk.dm-0.in‚5.3125 disk.dm-1.out‚ 487.41015625 disk.rbd0.out‚0* proc.memory.percent‚17.460918426513672$ disk./.usageGB‚1098.745693206787 disk.dm-1.in‚ 7.31640625 disk.rbd3.out‚0 disk.nvme0n1.in‚5.3125 disk.rbd4.out‚0 disk.sda1.out‚0 disk.sda.in‚ 7.3671875' proc.memory.availableMB‚ 336.5390625 network.recv‚ 3.725295e+06) disk./.usagePercent‚73.73774538014948 disk.sda3.in‚ 7.31640625! proc.memory.rssMB‚ 2860.796875 proc.cpu.threads‚17 network.sent‚ 2.7752212e+07 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125& disk.nvme0c0n1.out‚49245.662109375# memory_percent‚97.9459285736084 cpu‚25.30567508520691! disk.dm-0.out‚49245.662109375 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.sda.out‚ 487.44921875 disk.sda3.out‚ 487.41015625! disk.rbd2.out‚321.82666015625 disk.rbd3.in‚0 disk.rbd1.out‚ 5.27587890625 disk.rbd1.in‚0$ disk.nvme0n1.out‚49245.662109375 disk.sda1.in‚ 0.05078125m‹2„*‚(:… ÷äÔÉì¼x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æhCˆo…*‚(:ä ÷äÔÖ§üz! disk.dm-0.out‚49246.111328125 disk.sda2.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 3.732478e+06 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 487.84375$ memory_percent‚97.94740676879883 network.sent‚ 2.7861207e+07 disk.nvme0n1.in‚5.3125$ disk.nvme0n1.out‚49246.111328125 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚ 0.0390625 disk.drbd1777.in‚ 1.03515625 cpu‚25.305341155419228 disk.dm-1.out‚ 487.84375% disk./.usageGB‚1098.7447814941406 disk.dm-0.in‚5.3125 disk.dm-1.in‚ 7.31640625 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.sda.out‚ 487.8828125& proc.memory.availableMB‚ 336.296875 disk.rbd1.in‚0! disk.rbd2.out‚321.82666015625 disk.rbd0.in‚0& disk.nvme0c0n1.out‚49246.111328125( disk./.usagePercent‚73.7376841943488 disk.rbd1.out‚ 5.27587890625 disk.sda.in‚ 7.3671875! proc.memory.rssMB‚ 2860.796875* proc.memory.percent‚17.460918426513672’b/M†*‚(:„ œ÷äԵΟx gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.111' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 xDÎv‡*‚(:ë œ÷äÔåÖ»z* proc.memory.percent‚17.460918426513672 proc.cpu.threads‚17 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚ 7.31640625 disk.nvme0c0n1.in‚5.3125& proc.memory.availableMB‚ 336.546875 disk.sda3.out‚ 488.26171875 disk.rbd1.in‚0 disk.rbd1.out‚ 5.27587890625$ disk.nvme0n1.out‚49345.173828125 disk.rbd0.out‚0 disk.rbd3.in‚0) disk./.usagePercent‚73.73768624240907! disk.dm-0.out‚49345.173828125& disk.nvme0c0n1.out‚49345.173828125 network.recv‚ 3.736025e+06% disk./.usageGB‚1098.7448120117188 disk.sda1.out‚0 network.sent‚ 2.7866488e+07 disk.rbd4.in‚0 disk.sda.out‚ 488.30078125 disk.sda2.in‚0 disk.sda3.in‚ 7.31640625 disk.rbd2.in‚0 disk.sda.in‚ 7.3671875 disk.dm-1.out‚ 488.26171875 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0! proc.memory.rssMB‚ 2860.796875 cpu‚25.30496094878779 disk.dm-0.in‚5.3125! disk.rbd2.out‚323.50634765625 disk.rbd0.in‚0$ memory_percent‚97.94588088989258½Äº4ˆ*/ _wandbruntime‚10635  _runtime‚10635po ¯‰*à trainer/global_step‚10949 $ _timestamp‚1.788427164749472e+09 _runtime‚10635.49940094 _step‚899 ' lr-AdamW/pg1‚2.1571933833794784e-05 ' lr-AdamW/pg—-3,2‚2.1571933833794784e-05n®KΊ*È ' lr-AdamW/pg1‚2.1571933833794784e-05 ' lr-AdamW/pg2‚2.1571933833794784e-05 trainer/global_step‚10949 $ _timestamp‚1.788427164749472e+09 _runtime‚10635.49940094 _step‚899ƒIÖ,?4‹*/ _wandbruntime‚10635  _runtime‚10635Õ,îÃð'Œ*êO N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 192342226 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 202503917 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20288821784433 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 487744902 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚434639380350976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20347671958528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 202503915 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20700525510656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454987034306560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 487719230 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚411560056226816 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118481110 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 285215315 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 285240985 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20288598135285 trainer/global_step‚10949 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 285240985 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20347900162048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20347900162048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚487067492691968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431848878011249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚487067789647872 _step‚900 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚431848637322741 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454987034306560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚411560039187456ß;ŠHõ'*ïO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 192342912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20347671958528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚434639362348032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20700798526464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚10635.500291909 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454987280513024 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 487744902 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚434639380350976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20347671958528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 202503915 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20700525510656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454987034306560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 487719230 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚411560056226816 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118481110 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 285215315 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 285240985 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20288598135285 trainer/global_step‚10949 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 285240985 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20347900162048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20347900162048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296955904 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚487067492691968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431848878011249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚487067789647872 _step‚900 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚431848637322741 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454987034306560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚411560039187456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 192342226 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 202503917 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20288821784433 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 487762500 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚466408160817664 _step‚901 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 487762500 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚455027209945088 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192358792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20349737869824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20702662590976 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚434677718295040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431887012868913 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚455027456164864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20349737869824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚434677700292096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚657 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 487788198 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 202521636 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118491465 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118491463 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884271658378043e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 202521634 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚487110532757504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 285240866 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚10949 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20349509652992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚434677718295040 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 487788198 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 192359451 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚487110823408640 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299ÂÑçî˜*úM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20349737869824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20702662590976 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚434677718295040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431887012868913 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚455027456164864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20349737869824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚434677700292096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all5OÑòh.current‚659 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚657 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 487788198 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 202521636 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118491465 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118491463 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884271658378043e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 202521634 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚487110532757504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 285240866 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266711040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 trainer/global_step‚10949 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20349509652992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚434677718295040 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 487788198 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 192359451 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚487110823408640 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 285240866 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚466408136877568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 487762500 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚466408160817664 _step‚901 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 487762500 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚455027209945088 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192358792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561…Qf`4‘*/ _wandbruntime‚10636  _runtime‚10636¾j–b’*Ü 1loss/FBt2/assignment_loss‚0.02787558059238166 2loss/SRqqt2/assignment_loss‚0.8974986438535981 /loss/FRt2/detection_loss‚1.9090176604920999 1loss/SRqqt2/detection_loss‚0.9115159764575461 _step‚902 1loss/SRqqt1/detection_loss‚2.1069155090894456 &loss/total_loss‚40.377639344231326 _runtime‚10636.589503711 /loss/FRt1/assignment_loss‚2.869062160834288 0loss/FRt2/assignment_loss‚1.3976609364151955 2loss/FBt3/assignment_loss‚0.055664928957412485 /loss/FRt1/detection_loss‚2.5350883159881983 .loss/FRt3/detection_loss‚2.460555409876312 1loss/SRqqt4/detection_loss‚1.8538357317447662 /loss/FBt3/detection_loss‚2.4355531483888626 2loss/SRqqt4/assignment_loss‚1.2720655831305876 /loss/FBt1/detection_loss‚2.4185480535030366 % _timestamp‚1.7884271658396645e+09 2loss/FBt4/assignment_loss‚0.001532118224228422 .loss/FRt4/detection_loss‚2.458408521978599 /loss/FBt2/detection_loss‚0.6189262777167771 .loss/FBt4/detection_loss‚3.682714581489563 /loss/FRt3/assignment_loss‚2.462577422944511 0loss/FRt4/assignment_loss‚2.9324611676366708 2loss/SRqqt1/assignment_loss‚1.5009793623899803 1loss/SRqqt3/detection_loss‚1.8982852379912916 epoch‚9 trainer/global_step‚10949 2loss/SRqqt3/assignment_loss‚1.4346236052072567 0loss/FBt1/assignment_loss‚0.2362734093287145|ÓQg“*á /loss/FRt3/assignment_loss‚2.462577422944511 0loss/FRt4/assignment_loss‚2.9324611676366708 2loss/SRqqt1/assignment_loss‚1.5009793623899803 1loss/SRqqt3/detection_loss‚1.8982852379912916 epoch‚9 trainer/global_step‚10949 2loss/SRqqt3/assignment_loss‚1.4346236052072567 0loss/FBt1/assignment_loss‚0.2362734093287145 1loss/FBt2/assignment_loss‚0.02787558059238166 2loss/SRqqt2/assignment_loss‚0.8974986438535981 /loss/FRt2/detection_loss‚1.9090176604920999 1loss/SRqqt2/detection_loss‚0.9115159764575461 _step‚902 1loss/SRqqt1/detection_loss‚2.1069155090894456 &loss/total_loss‚40.377639344231326 _runtime‚10636.589503711 /loss/FRt1/assignment_loss‚2.869062160834288 0loss/FRt2/assignment_loss‚1.3976609364151955 2loss/FBt3/assignment_loss‚0.055664928957412485 /loss/FRt1/detection_loss‚2.5350883159881983 .loss/FRt3/detection_loss‚2.460555409876312 1loss/SRqqt4/detection_loss‚1.8538357317447662 /loss/FBt3/detection_loss‚2.4355531483888626 2loss/SRqqt4/assignment_loss‚1.2720655831305876 /loss/FBt1/detection_loss‚2.4185480535030366 % _timestamp‚1.7884271658396645e+09 2loss/FBt4/assignment_loss‚0.001532118224228422 .loss/FRt4/detection_loss‚2.458408521978599 /loss/FBt2/detection_loss‚0.6189262777167771 .loss/FBt4/detection_loss‚3.682714581489563†±Üüš”*‚(: «÷äÔø¦µx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.57' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.913999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gc]~•*‚(:ó «÷äÔ‰üâz( disk.nvme0c0n1.out‚49444.59814453125 disk.sda.in‚ 7.3671875 disk.sda3.in‚ 7.31640625 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0! disk.rbd2.out‚323.50634765625 disk.rbd3.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 488.98046875 disk.rbd4.in‚0' proc.memory.availableMB‚ 336.5703125 disk.rbd1.out‚ 5.27587890625& disk.nvme0n1.out‚49444.59814453125# disk.dm-0.out‚49444.59814453125 disk.sda2.in‚0 disk.rbd2.in‚0! proc.memory.rssMB‚ 2860.796875 proc.cpu.threads‚17 disk.sda3.out‚ 488.98046875 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125$ memory_percent‚97.94573783874512 network.sent‚ 2.7872749e+07) disk./.usagePercent‚73.73775178033783% disk./.usageGB‚1098.7457885742188 disk.drbd1777.out‚0 cpu‚25.304624629755168 network.recv‚ 3.740775e+06 disk.sda1.in‚ 0.05078125 disk.rbd0.out‚0* proc.memory.percent‚17.460918426513672 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.out‚ 489.01953125 disk.dm-1.in‚ 7.31640625pÌû…–*‚(:„ º÷äÔ±•«x gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žáqNx—*‚(:í º÷äÔÊÖŠ{$ memory_percent‚97.94886112213135 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.3671875 disk.sda2.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 336.05859375* proc.memory.percent‚17.460918426513672% disk./.usageGB‚1098.7457885742188 disk.rbd3.out‚0! disk.rbd2.out‚325.25634765625 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 489.44140625 network.recv‚ 3.747392e+06) disk./.usagePercent‚73.73775178033783 disk.sda1.out‚0! disk.dm-0.out‚49444.947265625 disk.rbd4.out‚0 disk.rbd1.out‚ 5.27587890625 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 cpu‚25.30424251044658 proc.cpu.threads‚17 network.sent‚ 2.7981271e+07 disk.rbd3.in‚0 disk.sda.out‚ 489.48046875 disk.dm-1.out‚ 489.44140625 disk.drbd1777.in‚ 1.03515625& disk.nvme0c0n1.out‚49444.947265625! proc.memory.rssMB‚ 2860.796875 disk.dm-1.in‚ 7.31640625 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 disk.nvme0c0n1.in‚5.3125$ disk.nvme0n1.out‚49444.947265625 disk.rbd2.in‚0Èø.„›˜*‚(: É÷äÔݾ˜y gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wbÖév™*‚(:ë É÷äÔÑãˆ| disk.sda3.out‚ 490.08203125! proc.memory.rssMB‚ 2860.796875* proc.memory.percent‚17.460918426513672% disk.nvme0n1.out‚49544.5244140625 disk.sda1.out‚0 disk.dm-1.out‚ 490.08203125 network.sent‚ 2.798763e+07% disk./.usageGB‚1098.7457962036133 disk.sda.in‚ 7.3671875 disk.rbd1.out‚ 5.27587890625 disk.nvme0n1.in‚5.3125 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 7.31640625 cpu‚25.30390481054875' disk.nvme0c0n1.out‚49544.5244140625( disk./.usagePercent‚73.7377522923529 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.rbd4.in‚0$ memory_percent‚97.95045852661133 disk.rbd0.out‚0 disk.sda.out‚ 490.12109375 disk.rbd1.in‚0" disk.dm-0.out‚49544.5244140625 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 proc.cpu.threads‚17! disk.rbd2.out‚326.83056640625 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125& proc.memory.availableMB‚ 335.796875 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚ 7.31640625 network.recv‚ 3.75224e+06 disk.rbd4.out‚0`~4š*/ _wandbruntime‚10680  _runtime‚10680ñ霗Ë›*Å _step‚903 ' lr-AdamW/pg1‚1.9738064579053086e-05 ' lr-AdamW/pg2‚1.9738064579053086e-05 trainer/global_step‚10999 % _timestamp‚1.7884272102153916e+09 _runtime‚10680.965320474lîtòÐœ*Ê trainer/global_step‚10999 % _timestamp‚1.7884272102153916e+09 _runtime‚10680.965320474 _step‚903 ' lr-AdamW/pg1‚1.9738064579053086e-05 ' lr-AdamW/pg2‚1.9738064579053086e-05‡:ÕKe4*/ _wandbruntime‚10680  _runtime‚106809fÒð'ž*êO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20439571095552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚468425475934208 $ _timestamp‚1.788427210216108e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436556277554176 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 286496217 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 193182967 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 286521887 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚456995830646784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20439799299072 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193182298 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 489886082 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 203389865 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20794375238656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489219541634048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433755393786749 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20380455355641 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433755634475257 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74183440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚436556259551232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚413375162080256 _runtime‚10680.966233354 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚456996076853248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118998858 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚456995830646784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 286521887 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚468425451994112 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489219851172864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚10999 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 286496217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436556259551232 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 203389865 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 203389867 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20439799299072 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚456996076853248 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118998860 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20380231706493 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 _step‚904šÚÔëŸ*ïO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436556277554176 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 286496217 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 193182967 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 286521887 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚456995830646784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20439799299072 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193182298 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 489886082 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 203389865 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20794375238656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489219541634048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433755393786749 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20380455355641 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433755634475257 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74183440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚436556259551232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚413375162080256 _runtime‚10680.966233354 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚456996076853248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118998858 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚456995830646784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 286521887 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚468425451994112 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489219851172864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚10999 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 286496217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436556259551232 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 203389865 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 203389867 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20439799299072 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚456996076853248 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118998860 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20380231706493 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 _step‚904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20439571095552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚468425475934208 $ _timestamp‚1.788427210216108e+09ˆ¿Öï4 */ _wandbruntime‚10681  _runtime‚10681Å4¸ú&¡*ôM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20382287755449 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚413411464538112 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 119009213 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20382064106093 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20796235063296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚457036252505088 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20441637006848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _runtime‚10681.836917492 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 489955050 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433793769332921 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚657 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 _step‚905 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚468466645630464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 trainer/global_step‚10999 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489262579556864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚436594597495296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489262880693760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚436594597495296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20441408790016 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 286547464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 489929352 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119009215 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20795957866496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚413411481577472 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20441637006848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788427211086247e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 286521768 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚457036252505088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 193199833 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193199174 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 286547464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74190618 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824/kûÿ&¢*ùM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚436594597495296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489262880693760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚436594597495296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20441408790016 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 286547464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 489929352 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119009215 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301136896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20795957866496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚413411481577472 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20441637006848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788427211086247e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 286521768 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚457036252505088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 193199833 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193199174 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 286547464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74190618 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20382287755449 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 277196800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚413411464538112 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 119009213 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20382064106093 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20796235063296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚457036252505088 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20441637006848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 _runtime‚10681.836917492 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 489955050 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433793769332921 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚657 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 _step‚905 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚468466645630464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 trainer/global_step‚10999 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489262579556864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262‰KÄÝ…4£*/ _wandbruntime‚10681  _runtime‚10681¡'Wþ[¤*Õ 0loss/FRt2/assignment_loss‚1.2065605730341191 1loss/SRqqt2/assignment_loss‚1.066704707460538 /loss/FBt1/detection_loss‚2.1657692790031433 %loss/total_loss‚39.94509885508688 trainer/global_step‚10999 0loss/FRt3/assignment_loss‚2.3533036708831787 0loss/SRqqt1/detection_loss‚2.375089137752851 2loss/SRqqt1/assignment_loss‚1.5915167527894178 2loss/SRqqt4/assignment_loss‚1.7393624617956405 0loss/FBt2/assignment_loss‚0.1290778906597403 -loss/FRt2/detection_loss‚1.43386203089827 % _timestamp‚1.7884272110881822e+09 1loss/FBt1/assignment_loss‚0.41862729452259373 .loss/FRt4/detection_loss‚2.416066326419167 /loss/FBt3/detection_loss‚1.9598530105182104 _step‚906 /loss/FRt4/assignment_loss‚3.494215840952737 /loss/FRt3/detection_loss‚2.5832131779507588 1loss/SRqqt2/detection_loss‚1.0143077486465055 1loss/SRqqt3/detection_loss‚1.6843712904623576 epoch‚9 _runtime‚10681.83802099 1loss/FBt4/assignment_loss‚0.20824446697952226 /loss/FRt1/assignment_loss‚3.020230056511031 2loss/SRqqt3/assignment_loss‚1.4028670921617625 1loss/FBt3/assignment_loss‚0.14137515050658425 .loss/FBt4/detection_loss‚2.389964830875397 /loss/FRt1/detection_loss‚2.5934389241867595 1loss/SRqqt4/detection_loss‚2.0498706102371216 /loss/FBt2/detection_loss‚0.5072065298794769X„}`¥*Ú 1loss/SRqqt4/detection_loss‚2.0498706102371216 /loss/FBt2/detection_loss‚0.5072065298794769 0loss/FRt2/assignment_loss‚1.2065605730341191 1loss/SRqqt2/assignment_loss‚1.066704707460538 /loss/FBt1/detection_loss‚2.1657692790031433 %loss/total_loss‚39.94509885508688 trainer/global_step‚10999 0loss/FRt3/assignment_loss‚2.3533036708831787 0loss/SRqqt1/detection_loss‚2.375089137752851 2loss/SRqqt1/assignment_loss‚1.5915167527894178 2loss/SRqqt4/assignment_loss‚1.7393624617956405 0loss/FBt2/assignment_loss‚0.1290778906597403 -loss/FRt2/detection_loss‚1.43386203089827 % _timestamp‚1.7884272110881822e+09 1loss/FBt1/assignment_loss‚0.41862729452259373 .loss/FRt4/detection_loss‚2.416066326419167 /loss/FBt3/detection_loss‚1.9598530105182104 _step‚906 /loss/FRt4/assignment_loss‚3.494215840952737 /loss/FRt3/detection_loss‚2.5832131779507588 1loss/SRqqt2/detection_loss‚1.0143077486465055 1loss/SRqqt3/detection_loss‚1.6843712904623576 epoch‚9 _runtime‚10681.83802099 1loss/FBt4/assignment_loss‚0.20824446697952226 /loss/FRt1/assignment_loss‚3.020230056511031 2loss/SRqqt3/assignment_loss‚1.4028670921617625 1loss/FBt3/assignment_loss‚0.14137515050658425 .loss/FBt4/detection_loss‚2.389964830875397 /loss/FRt1/detection_loss‚2.5934389241867595Š¤OÑߦ*‚(:… Ø÷äÔ—ºx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.011' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02yÊ[|§*‚(:ñ Ø÷äÔ—¡ðz disk.sda.in‚ 7.3671875 disk.sda.out‚ 490.54296875 disk.sda3.out‚ 490.50390625 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚327.0029296875 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚49643.97216796875 network.sent‚ 2.799291e+07 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 5.27587890625 disk.dm-1.out‚ 490.50390625 disk.rbd2.in‚0" proc.memory.rssMB‚ 2774.9140625 cpu‚25.30353208969121 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.rbd1.in‚0# disk.dm-0.out‚49643.97216796875 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚5.3125* proc.memory.percent‚16.936731338500977( disk./.usagePercent‚73.7377522923529 disk.sda3.in‚ 7.31640625 disk.dm-0.in‚5.3125 proc.cpu.threads‚17% disk./.usageGB‚1098.7457962036133 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0& disk.nvme0n1.out‚49643.97216796875 disk.dm-1.in‚ 7.31640625$ memory_percent‚97.42496013641357( proc.memory.availableMB‚ 421.89453125 network.recv‚ 3.755627e+06 disk.rbd3.in‚0Ž3$¨*‚(:… ç÷äÔÞüw gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.149' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶µW²y©*‚(:î ç÷äÔ£³{) proc.memory.percent‚16.93868637084961) disk./.usagePercent‚73.73775178033783" disk.dm-0.out‚49743.6279296875 disk.sda.out‚ 491.17578125% disk./.usageGB‚1098.7457885742188 disk.dm-1.out‚ 491.13671875 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 5.27587890625 disk.nvme0n1.in‚5.3125% disk.nvme0n1.out‚49743.6279296875 network.recv‚ 3.761903e+06' disk.nvme0c0n1.out‚49743.6279296875 disk.rbd1.in‚0$ memory_percent‚97.42541313171387! proc.memory.rssMB‚ 2775.234375 cpu‚25.303176721370072 proc.cpu.threads‚17 disk.dm-1.in‚ 7.31640625 disk.rbd4.in‚0 disk.rbd3.out‚0 network.sent‚ 2.8100707e+07 disk.rbd2.in‚0 disk.rbd2.out‚328.5771484375 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.sda3.out‚ 491.13671875 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.in‚ 7.3671875' proc.memory.availableMB‚ 421.8203125 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 7.31640625F´ÊÖª*‚(:… ö÷äÔŠ¹šx gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}õÑŠm«*‚(:â ö÷äÔÝ´îz disk.rbd3.out‚0 network.recv‚ 3.765926e+06 disk.nvme0c0n1.in‚5.3125 disk.sda3.out‚ 491.58203125 disk.dm-1.in‚ 7.31640625 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.out‚0( proc.memory.availableMB‚ 421.83203125 disk.rbd1.out‚ 5.27587890625 disk.rbd2.in‚0) disk./.usagePercent‚73.73775178033783 disk.sda3.in‚ 7.31640625$ memory_percent‚97.42534160614014% disk./.usageGB‚1098.7457885742188 disk.dm-0.out‚ 49743.984375 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 49743.984375 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 491.62109375 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.dm-1.out‚ 491.58203125 disk.rbd2.out‚328.6865234375 disk.rbd4.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2775.234375 network.sent‚ 2.8106581e+07 disk.sda.in‚ 7.3671875 disk.rbd3.in‚0) proc.memory.percent‚16.93868637084961 cpu‚25.30282474679119 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125! disk.nvme0n1.out‚ 49743.984375 disk.rbd0.in‚0¾zy4¬*/ _wandbruntime‚10726  _runtime‚10726³A¥ É­*à & lr-AdamW/pg2‚1.798368294548949e-05 trainer/global_step‚11049 % _timestamp‚1.7884272554787855e+09 _runtime‚10726.228716887 _step‚907 & lr-AdamW/pg1‚1.798368294548949e-05A'—ή*È & lr-AdamW/pg1‚1.798368294548949e-05 & lr-AdamW/pg2‚1.798368294548949e-05 trainer/global_step‚11049 % _timestamp‚1.7884272554787855e+09 _runtime‚10726.228716887 _step‚907‹JHü4¯*/ _wandbruntime‚10726  _runtime‚10726”îQÏh°*ëO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 287802789 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 287802789 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 287777119 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_W{‰$pool.allocated‚ 119516610 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚10726.229657479 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚459004626987008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 492078606 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204275817 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚435662390939265 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 trainer/global_step‚11049 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚415190302012416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚74506795 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚491371502683648 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 204275817 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚415190284973056 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚459004873193472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 204275815 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚438473156754432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚491371793348096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20531470232576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20887565876736 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824ËÁ~ö'±*ðO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 287777119 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20531698436096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74506172 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚459004626987008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20471865277701 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 194022780 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚435662150250757 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚438473174757376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119516608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚470483936806912 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20472088926849 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚438473174757376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 194023405 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚415190284973056 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚459004873193472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 204275815 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚438473156754432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚491371793348096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20531470232576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20887565876736 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 287802789 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 287802789 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 287777119 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 119516610 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚10726.229657479 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚459004626987008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 492078606 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204275817 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚435662390939265 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 trainer/global_step‚11049 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚415190302012416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚74506795 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚491371502683648 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 204275817 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚435700285108213 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 204293534 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11049 _step‚909 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 492121902 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚459045048845312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74512419 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20473697677301 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 204293536 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚459044802625536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚415226587430912 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚670 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119526965 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 492096204 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20473921326657 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438511512701440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 492096204 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20533307927040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚438511494698496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 204293536 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20533536143872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 287802670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20533536143872 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20889431853568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74513089 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 287828366 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚10727.090007461 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚491414538356736 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 % _timestamp‚1.7884272563392942e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194039382 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚470525106503168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚415226604470272 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561z }e/ ´*úM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 204293534 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚459045048845312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 287802670 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚438511512701440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 194040054 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20889704856064 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 287828366 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚435700285108213 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 204293534 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11049 _step‚909 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 492121902 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚459045048845312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74512419 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20473697677301 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 204293536 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚459044802625536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚415226587430912 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚670 C=DeviceStatsMonitor.on_train_batch_end/oversize_segmen‘¢]6Ñts.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119526965 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 492096204 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20473921326657 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438511512701440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 492096204 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20533307927040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚438511494698496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 204293536 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20533536143872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 287802670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20533536143872 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20889431853568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74513089 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 287828366 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚10727.090007461 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚491414538356736 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 % _timestamp‚1.7884272563392942e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194039382 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚470525106503168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚415226604470272 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 492121902Á/©4µ*/ _wandbruntime‚10727  _runtime‚10727ˆ`‰^¶*Ø 2loss/SRqqt1/assignment_loss‚1.9356549908133114 0loss/FBt1/assignment_loss‚0.6100403780195242 .loss/FRt3/detection_loss‚2.288733836263418 1loss/SRqqt1/detection_loss‚2.1566771496744717 1loss/SRqqt2/detection_loss‚0.9074589810264644 0loss/SRqqt3/detection_loss‚1.826783373620775 0loss/SRqqt4/detection_loss‚2.120333229833179 0loss/FBt2/detection_loss‚0.49515333023543157 /loss/FRt1/detection_loss‚2.4286644798378614 /loss/FBt1/detection_loss‚2.3485914369424186 /loss/FBt4/detection_loss‚4.1449199467897415 trainer/global_step‚11049 _step‚910 0loss/FRt2/assignment_loss‚1.5102426889869902 .loss/FRt2/detection_loss‚1.947930696937773 2loss/SRqqt3/assignment_loss‚1.1492694256206353 2loss/SRqqt4/assignment_loss‚1.5804428021113077 .loss/FBt3/detection_loss‚2.259900172551473 %loss/total_loss‚43.14839909828566 /loss/FRt1/assignment_loss‚3.394913074582122 1loss/FBt3/assignment_loss‚0.06665524104998137 1loss/FBt2/assignment_loss‚0.08627217613034371 3loss/FBt4/assignment_loss‚0.0014256778122216929 epoch‚9 0loss/FRt3/assignment_loss‚3.1625056018431983 /loss/FRt4/assignment_loss‚3.599985040369488 2loss/SRqqt2/assignment_loss‚0.8374383585499738 .loss/FRt4/detection_loss‚2.288407008683557 $ _timestamp‚1.788427256341215e+09 _runtime‚10727.091082345% âc·*Ý 0loss/SRqqt4/detection_loss‚2.120333229833179 0loss/FBt2/detection_loss‚0.49515333023543157 /loss/FRt1/detection_loss‚2.4286644798378614 /loss/FBt1/detection_loss‚2.3485914369424186 /loss/FBt4/detection_loss‚4.1449199467897415 trainer/global_step‚11049 _step‚910 0loss/FRt2/assignment_loss‚1.5102426889869902 .loss/FRt2/detection_loss‚1.947930696937773 2loss/SRqqt3/assignment_loss‚1.1492694256206353 2loss/SRqqt4/assignment_loss‚1.5804428021113077 .loss/FBt3/detection_loss‚2.259900172551473 %loss/total_loss‚43.14839909828566 /loss/FRt1/assignment_loss‚3.394913074582122 1loss/FBt3/assignment_loss‚0.06665524104998137 1loss/FBt2/assignment_loss‚0.08627217613034371 3loss/FBt4/assignment_loss‚0.0014256778122216929 epoch‚9 0loss/FRt3/assignment_loss‚3.1625056018431983 /loss/FRt4/assignment_loss‚3.599985040369488 2loss/SRqqt2/assignment_loss‚0.8374383585499738 .loss/FRt4/detection_loss‚2.288407008683557 $ _timestamp‚1.788427256341215e+09 _runtime‚10727.091082345 2loss/SRqqt1/assignment_loss‚1.9356549908133114 0loss/FBt1/assignment_loss‚0.6100403780195242 .loss/FRt3/detection_loss‚2.288733836263418 1loss/SRqqt1/detection_loss‚2.1566771496744717 1loss/SRqqt2/detection_loss‚0.9074589810264644 0loss/SRqqt3/detection_loss‚1.826783373620775ŽíR«u¸*‚(:„ …øäÔ﮵x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚91.181' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò¡]o¹*‚(:ä …øäÔ–ëÏ{ disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda3.in‚ 7.31640625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0) proc.memory.percent‚16.93868637084961% disk./.usageGB‚1098.7457885742188 disk.dm-0.in‚5.3125 disk.rbd0.out‚0! proc.memory.rssMB‚ 2775.234375 disk.rbd1.in‚0 cpu‚25.30246882304264 proc.cpu.threads‚17& disk.nvme0n1.out‚49843.54150390625 disk.sda3.out‚ 492.09375$ memory_percent‚97.42584228515625 network.recv‚ 3.770792e+06) disk./.usagePercent‚73.73775178033783 disk.nvme0n1.in‚5.3125 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.3671875 disk.sda.out‚ 492.1328125 network.sent‚ 2.8112988e+07 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚49843.54150390625 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0" proc.memory.availableMB‚421.75 disk.rbd1.out‚ 5.36181640625# disk.dm-0.out‚49843.54150390625 disk.dm-1.in‚ 7.31640625 disk.rbd2.out‚330.2607421875 disk.sda2.in‚0 disk.dm-1.out‚ 492.09375 disk.rbd2.in‚0Œ.—›º*‚(: ”øäÔµ±–x gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.775' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.755000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ s»*‚(:è ”øäÔ¼êÅz disk.rbd0.out‚0' disk.nvme0c0n1.out‚49945.7255859375 disk.sda1.out‚0 disk.sda3.in‚ 7.31640625 disk.dm-0.in‚5.3125 disk.rbd4.in‚0$ memory_percent‚97.42586612701416 proc.cpu.threads‚17% disk./.usageGB‚1098.7485847473145 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd4.out‚0 disk.sda.out‚ 492.6328125 disk.dm-1.out‚ 492.59375 disk.nvme0c0n1.in‚5.3125! proc.memory.rssMB‚ 2775.234375) proc.memory.percent‚16.93868637084961" disk.dm-0.out‚49945.7255859375 cpu‚25.302099614373443) disk./.usagePercent‚73.73793943386043 disk.drbd1777.in‚ 1.03515625 disk.rbd3.in‚0 disk.rbd3.out‚0 network.recv‚ 3.776928e+06 disk.sda3.out‚ 492.59375 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125( proc.memory.availableMB‚ 421.74609375 disk.nvme0n1.in‚5.3125 disk.dm-1.in‚ 7.31640625 disk.rbd2.out‚330.4873046875 disk.drbd1777.out‚0 disk.sda.in‚ 7.3671875 network.sent‚ 2.8220988e+07 disk.rbd1.in‚0 disk.rbd1.out‚ 5.42431640625% disk.nvme0n1.out‚49945.7255859375¼zò¼*‚(:… £øäÔàü˜x gpu.0.gpu‚20.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ürÎw½*‚(:ì £øäÔûô’{ network.sent‚ 2.8226937e+07 disk.sda3.out‚ 493.1953125 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 493.234375! proc.memory.rssMB‚ 2775.234375 disk.dm-0.in‚5.3125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 421.4921875 disk.sda3.in‚ 7.31640625 disk.dm-1.in‚ 7.31640625 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚50045.13232421875) disk./.usagePercent‚73.73794096990564 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚50045.13232421875 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 493.1953125 cpu‚25.301769563911122 disk.sda1.in‚ 0.05078125& disk.nvme0n1.out‚50045.13232421875 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.in‚ 7.3671875$ memory_percent‚97.42741584777832) proc.memory.percent‚16.93868637084961 network.recv‚ 3.781523e+06$ disk./.usageGB‚1098.748607635498 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 5.42431640625 disk.rbd2.out‚331.9990234375\óÿâ4¾*/ _wandbruntime‚10771  _runtime‚10771 *ÐË¿*Å ' lr-AdamW/pg1‚1.6309154543606905e-05 ' lr-AdamW/pg2‚1.6309154543606905e-05 trainer/global_step‚11099 % _timestamp‚1.7884273009083622e+09 _runtime‚10771.658325862 _step‚911‡ë+ÐÀ*Ê _step‚911 ' lr-AdamW/pg1‚1.6309154543606905e-05 ' lr-AdamW/pg2‚1.6309154543606905e-05 trainer/global_step‚11099 % _timestamp‚1.7884273009083622e+09 _runtime‚10771.658325862Z¥Zw4Á*/ _wandbruntime‚10771  _runtime‚10771pªÇñ'Â*ëO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74830703 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚493523406054912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚472542421619712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 205161765 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚461013669533696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20623597573120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120034358 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 205161767 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 289083691 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 194865696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884273009090934e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚440390053957632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20981255353856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 494219786 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120034360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚440390071960576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚461013669533696 _runtime‚10771.659258589 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚461013423327232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚493523700913664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 289058021 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚461013423327232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 494245458 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚417005407865856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚417005424905216 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚440390053957632 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 205161767 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20980984435200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528óŸKÃÃ*ðO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 294858752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20563498848909 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _step‚912 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚440390053957632 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 205161767 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20980984435200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74830703 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚493523406054912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚472542421619712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 205161765 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚461013669533696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20623597573120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120034358 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 205161767 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 289083691 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 194865696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884273009090934e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚440390053957632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20981255353856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 494219786 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120034360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚440390071960576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚461013669533696 _runtime‚10771.659258589 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚461013423327232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚493523700913664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 289058021 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚461013423327232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 494245458 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚417005407865856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚417005424905216 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚461053845185536 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 205179484 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 289109268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74837917 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 205179486 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚440428409904640 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120044715 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120044713 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20625207064064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 _step‚913 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 494288754 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚472583615256064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚417041727363072 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚417041710323712 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437607282260937 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 289083572 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 289083572 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚440428409904640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20625435280896 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20565331248509 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚461053845185536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 494288754 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20983128269824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884273017806492e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 205179484 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 trainer/global_step‚11099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 494263056 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚493566743525888 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20982846878720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 205179486 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚440428391901696 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194881989 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 494263056 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚10772.531389257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832ò'Æ*úM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚10772.531389257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚461053845185536 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 205179484 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 289109268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74837917 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 205179486 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚440428409904640 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120044715 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120044713 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20625207064064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 _step‚913 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 494288754 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚472583615256064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚417041727363072 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚417041710323712 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437607282260937 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 289083572 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 289083572 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚440428409904640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20625435280896 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20565331248509 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚461053845185536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 494288754 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20983128269824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884273017806492e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 205179484 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 trainer/global_step‚11099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 494263056 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚493566743525888 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20982846878720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 205179486 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚440428391901696 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194881989 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 494263056 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚439475663178773 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20715496710144 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 195708434 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚474600930372608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚463022219667456 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚75156326 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚463022219667456 trainer/global_step‚11149 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚418820530758656 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442306951160832 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120552108 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 _step‚916 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚463022465873920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 206047715 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120552110 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚474600906432512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960wÛ2“ö'Õ*ðO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 290364593 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20715496710144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚439475903867281 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495675389601280 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚463022465873920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21074483168768 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 290338923 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 _runtime‚10817.294706882 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚439475663178773 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20715496710144 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 195708434 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚474600930372608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚463022219667456 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚75156326 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚463022219667456 trainer/global_step‚11149 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚418820530758656 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442306951160832 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120552108 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 _step‚916 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚463022465873920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 206047715 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120552110 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚474600906432512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 % _timestamp‚1.7884273465445185e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 206047717 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442306969163776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442306951160832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 195709082 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 496386638 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 206047715 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 206047717 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚495675688654336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992”d'†ã4Ö*/ _wandbruntime‚10818  _runtime‚10818èü°Ïû&×*õM YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20656964819717 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚418856833216512 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20657188469073 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚917 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚463062641525760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚442345289104896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚439514038724945 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚463062641525760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 496429908 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 290364474 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20717334417920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚75162539 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195725002 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚474642076128768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195725625 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _runtime‚10818.172470072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439513798036229 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 496429908 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 496455606 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 290364474 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120562463 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚11149 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442345307107840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20717334417920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 206065436 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 % _timestamp‚1.7884273474218018e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚442345289104896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75163160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚442345307107840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561àw‡ÚØ*úM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚418856833216512 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20657188469073 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚917 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚463062641525760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚442345289104896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚439514038724945 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚463062641525760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 496429908 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 290364474 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20717334417920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 REDeviceStaU=¿½&tsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚75162539 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195725002 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚474642076128768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195725625 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _runtime‚10818.172470072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439513798036229 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 496429908 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 496455606 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 290364474 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120562463 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚11149 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442345307107840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20717334417920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 206065436 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 % _timestamp‚1.7884273474218018e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚442345289104896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75163160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚442345307107840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20656964819717 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2•ƒtï4Ù*/ _wandbruntime‚10818  _runtime‚10818‹€/;\Ú*Ö .loss/FRt4/assignment_loss‚2.87310486247665 1loss/SRqqt3/assignment_loss‚1.225738462085636 /loss/FBt1/detection_loss‚1.9435616639944224 _step‚918 1loss/FBt1/assignment_loss‚0.19498509698902267 1loss/FBt4/assignment_loss‚0.07751081176102162 0loss/FBt2/detection_loss‚0.43257350815898354 epoch‚9 /loss/FRt2/detection_loss‚1.5500261529671902 .loss/FRt4/detection_loss‚2.510959361728979 1loss/SRqqt2/detection_loss‚1.0588243215211801 1loss/SRqqt3/detection_loss‚1.7748696559241839 /loss/FRt2/assignment_loss‚1.257812998732742 1loss/SRqqt2/assignment_loss‚0.764591161298403 1loss/FBt2/assignment_loss‚0.14724267714807127 /loss/FRt1/detection_loss‚2.6919575697845883 _runtime‚10818.173579292 2loss/SRqqt4/assignment_loss‚1.8335795171704947 .loss/FBt4/detection_loss‚2.693566620349884 % _timestamp‚1.7884273474237392e+09 .loss/FRt3/assignment_loss‚2.13827874759833 2loss/SRqqt1/assignment_loss‚1.4764190779533237 1loss/SRqqt4/detection_loss‚1.8341127146108478 %loss/total_loss‚38.94172248745875 trainer/global_step‚11149 0loss/FRt1/assignment_loss‚2.8316920151313147 .loss/FRt3/detection_loss‚2.800121895141072 1loss/SRqqt1/detection_loss‚2.4222709676250815 /loss/FBt3/detection_loss‚2.1649285554885864 1loss/FBt3/assignment_loss‚0.24299407181873298†ŸËaÛ*Û 1loss/SRqqt3/detection_loss‚1.7748696559241839 /loss/FRt2/assignment_loss‚1.257812998732742 1loss/SRqqt2/assignment_loss‚0.764591161298403 1loss/FBt2/assignment_loss‚0.14724267714807127 /loss/FRt1/detection_loss‚2.6919575697845883 _runtime‚10818.173579292 2loss/SRqqt4/assignment_loss‚1.8335795171704947 .loss/FBt4/detection_loss‚2.693566620349884 % _timestamp‚1.7884273474237392e+09 .loss/FRt3/assignment_loss‚2.13827874759833 2loss/SRqqt1/assignment_loss‚1.4764190779533237 1loss/SRqqt4/detection_loss‚1.8341127146108478 %loss/total_loss‚38.94172248745875 trainer/global_step‚11149 0loss/FRt1/assignment_loss‚2.8316920151313147 .loss/FRt3/detection_loss‚2.800121895141072 1loss/SRqqt1/detection_loss‚2.4222709676250815 /loss/FBt3/detection_loss‚2.1649285554885864 1loss/FBt3/assignment_loss‚0.24299407181873298 .loss/FRt4/assignment_loss‚2.87310486247665 1loss/SRqqt3/assignment_loss‚1.225738462085636 /loss/FBt1/detection_loss‚1.9435616639944224 _step‚918 1loss/FBt1/assignment_loss‚0.19498509698902267 1loss/FBt4/assignment_loss‚0.07751081176102162 0loss/FBt2/detection_loss‚0.43257350815898354 epoch‚9 /loss/FRt2/detection_loss‚1.5500261529671902 .loss/FRt4/detection_loss‚2.510959361728979 1loss/SRqqt2/detection_loss‚1.0588243215211801–¦ØdÜ*‚(:… ßøäÔé«ây gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰wµ6kÝ*‚(:à ßøäÔ•´Ùz disk.rbd3.out‚0 disk.sda3.out‚ 495.65625! proc.memory.rssMB‚ 2775.234375 cpu‚25.30035730719491) disk./.usagePercent‚73.73789386451935% disk./.usageGB‚1098.7479057312012 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.in‚ 7.3671875! disk.dm-0.out‚50244.498046875 disk.sda2.in‚0 disk.dm-1.in‚ 7.31640625 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚ 7.31640625 disk.rbd4.out‚0 network.sent‚ 2.8357176e+07 disk.nvme0n1.in‚5.3125 disk.rbd2.out‚335.4169921875 disk.drbd1777.in‚ 1.03515625 disk.dm-1.out‚ 495.65625 disk.sda.out‚ 495.6953125( proc.memory.availableMB‚ 421.42578125$ disk.nvme0n1.out‚50244.498046875) proc.memory.percent‚16.93868637084961 disk.dm-0.in‚5.3125 disk.rbd3.in‚0# memory_percent‚97.4278211593628 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 5.51953125 disk.rbd0.out‚0 disk.rbd4.in‚0 proc.cpu.threads‚17 network.recv‚ 3.808579e+06 disk.rbd0.in‚0& disk.nvme0c0n1.out‚50244.498046875OÍ…ŽÞ*‚(:ƒ îøäÔ‰—Îx gpu.0.gpu‚20.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bÂcxß*‚(:í îøäÔ½ Ùz% disk./.usageGB‚1098.7479057312012( disk.nvme0c0n1.out‚50343.89306640625 disk.rbd3.out‚0# disk.dm-0.out‚50343.89306640625 disk.dm-1.in‚ 7.31640625 network.recv‚ 3.815648e+06 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 5.58251953125 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.sda3.in‚ 7.31640625 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd2.out‚337.0341796875 disk.rbd4.in‚0 disk.sda3.out‚ 496.109375 disk.sda2.in‚0 network.sent‚ 2.8465949e+07 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda.in‚ 7.3671875 disk.dm-0.in‚5.3125$ memory_percent‚97.42658138275146 disk.nvme0c0n1.in‚5.3125 disk.dm-1.out‚ 496.109375 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0& disk.nvme0n1.out‚50343.89306640625) disk./.usagePercent‚73.73789386451935 disk.rbd3.in‚0 disk.sda.out‚ 496.1484375( proc.memory.availableMB‚ 421.62890625! proc.memory.rssMB‚ 2775.234375) proc.memory.percent‚16.93868637084961 cpu‚25.300011852746092=ZüHšà*‚(: ýøäÔ¦žx gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.57' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.913999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«H|qá*‚(:æ ýøäÔáñÍ{ disk.rbd3.in‚0 disk.sda.out‚ 496.6953125 cpu‚25.29964005179208 network.recv‚ 3.820738e+06) disk./.usagePercent‚73.73791050500907 disk.rbd1.out‚ 5.58251953125 disk.dm-1.in‚ 7.31640625% disk./.usageGB‚1098.7481536865234 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 disk.rbd2.in‚0' disk.nvme0c0n1.out‚50443.4384765625 disk.sda3.out‚ 496.65625 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125) proc.memory.percent‚16.93868637084961 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.in‚ 7.3671875 network.sent‚ 2.847228e+07" disk.dm-0.out‚50443.4384765625 disk.sda3.in‚ 7.31640625 disk.nvme0n1.in‚5.3125% disk.nvme0n1.out‚50443.4384765625 disk.dm-1.out‚ 496.65625 disk.rbd2.out‚338.5458984375 disk.rbd3.out‚0( proc.memory.availableMB‚ 421.62890625 proc.cpu.threads‚17$ memory_percent‚97.42658138275146! proc.memory.rssMB‚ 2775.234375 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.050781255®øH4â*/ _wandbruntime‚10862  _runtime‚10862øéàúÊã*Ä trainer/global_step‚11199 $ _timestamp‚1.788427392172013e+09 _runtime‚10862.921980625 _step‚919 ' lr-AdamW/pg1‚1.3201036597702768e-05 ' lr-AdamW/pg2‚1.3201036597702768e-05éùäÏä*É $ _timestamp‚1.788427392172013e+09 _runtime‚10862.921980625 _step‚919 ' lr-AdamW/pg1‚1.3201036597702768e-05 ' lr-AdamW/pg2‚1.3201036597702768e-05 trainer/global_step‚11199—~ÓS4å*/ _wandbruntime‚10862  _runtime‚10862+"òÐð'æ*êO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚444223866366976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20807167643648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20807395847168 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 121069860 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196549678 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 _step‚920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21167922856960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465031016007680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 498553490 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 206933665 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚420635653651456 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 206933667 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚444223848364032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚444223848364032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚497827613155328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚476659415185408 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299053056 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 498579162 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 291645495 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465031016007680 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20807395847168 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 206933665 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21168197969920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 291619825 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚476659391245312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 291645495 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465031016007680 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20807395847168 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 206933665 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21168197969920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 291619825 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚476659391245312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275112960 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20807167643648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20807395847168 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 121069860 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196549678 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 _step‚920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21167922856960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624˜‡k4è*/ _wandbruntime‚10863  _runtime‚10863$i[û&é*õM F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚497870665534464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚497870381174784 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20748822040281 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 291671072 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 206951384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 498622458 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 291671072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884273930285566e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚444262186308096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21170080652800 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196565620 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20809233554944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465071437865984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 284359680 _step‚921 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441420795188953 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 206951386 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 291645376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465071191646208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 260419584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 498596760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20809233554944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75486031 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚476700560941568 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465071437865984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 498596760 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚441420554500237 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚420671973148672 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚444262204311040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20809005338112 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚11199 _runtime‚10863.779158876 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20748598390925 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21169820233216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624BÎÅ'ê*úM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚11199 _runtime‚10863.779158876 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20748598390925 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21169820233216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚497870665534464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚497870381174784 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20748822040281 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 291671072 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 206951384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 498622458 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 291671072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 % _timestamp‚1.7884273930285566e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚444262186308096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21170080652800 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196565620 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20809233554944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465071437865984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 284359680 _step‚921 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441420795188953 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 206951386 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 291645376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465071191646208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 260419584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 498596760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20809233554944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75486031 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚476700560941568 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465071437865984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 498596760 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚441420554500237 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚420671973148672 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚444262204311040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20809005338112 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832™smY4ë*/ _wandbruntime‚10863  _runtime‚10863Õé÷Zcì*Ý 1loss/FBt4/assignment_loss‚0.14936526915213713 epoch‚9 _runtime‚10863.780272463 0loss/FRt1/assignment_loss‚2.7932166180959563 0loss/FRt4/assignment_loss‚2.4083339431706596 1loss/FBt1/assignment_loss‚0.19823556739328735 /loss/FBt1/detection_loss‚2.3139890340658336 /loss/FBt4/detection_loss‚3.1214123964309692 trainer/global_step‚11199 0loss/FRt3/assignment_loss‚2.4638137925754893 /loss/FBt2/detection_loss‚0.5175478456434436 0loss/FRt2/assignment_loss‚1.0107621583071622 2loss/SRqqt1/assignment_loss‚1.3846710728274452 2loss/SRqqt2/assignment_loss‚0.9883791330601956 2loss/SRqqt3/assignment_loss‚0.9072735472023488 .loss/FRt4/detection_loss‚2.407494494152542 0loss/SRqqt4/detection_loss‚1.849459686378638 % _timestamp‚1.7884273930304558e+09 _step‚922 1loss/FBt2/assignment_loss‚0.12594052759226138 /loss/FRt1/detection_loss‚2.5583029825512957 /loss/FRt3/detection_loss‚2.4273064387115566 2loss/FBt3/assignment_loss‚0.013234277255833149 1loss/SRqqt2/detection_loss‚0.8207508982869528 /loss/FBt3/detection_loss‚3.1817557364702225 /loss/FRt2/detection_loss‚1.5076002162288535 0loss/SRqqt1/detection_loss‚2.226847003565894 %loss/total_loss‚38.61149074957177 2loss/SRqqt4/assignment_loss‚1.4566657102977236 1loss/SRqqt3/detection_loss‚1.7791324001550675Îu¾ƒhí*â 2loss/SRqqt1/assignment_loss‚1.3846710728274452 2loss/SRqqt2/assignment_loss‚0.9883791330601956 2loss/SRqqt3/assignment_loss‚0.9072735472023488 .loss/FRt4/detection_loss‚2.407494494152542 0loss/SRqqt4/detection_loss‚1.849459686378638 % _timestamp‚1.7884273930304558e+09 _step‚922 1loss/FBt2/assignment_loss‚0.12594052759226138 /loss/FRt1/detection_loss‚2.5583029825512957 /loss/FRt3/detection_loss‚2.4273064387115566 2loss/FBt3/assignment_loss‚0.013234277255833149 1loss/SRqqt2/detection_loss‚0.8207508982869528 /loss/FBt3/detection_loss‚3.1817557364702225 /loss/FRt2/detection_loss‚1.5076002162288535 0loss/SRqqt1/detection_loss‚2.226847003565894 %loss/total_loss‚38.61149074957177 2loss/SRqqt4/assignment_loss‚1.4566657102977236 1loss/SRqqt3/detection_loss‚1.7791324001550675 1loss/FBt4/assignment_loss‚0.14936526915213713 epoch‚9 _runtime‚10863.780272463 0loss/FRt1/assignment_loss‚2.7932166180959563 0loss/FRt4/assignment_loss‚2.4083339431706596 1loss/FBt1/assignment_loss‚0.19823556739328735 /loss/FBt1/detection_loss‚2.3139890340658336 /loss/FBt4/detection_loss‚3.1214123964309692 trainer/global_step‚11199 0loss/FRt3/assignment_loss‚2.4638137925754893 /loss/FBt2/detection_loss‚0.5175478456434436 0loss/FRt2/assignment_loss‚1.0107621583071622šÿ<0î*‚(:„ ŒùäԤћx gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä´/sqï*‚(:æ ŒùäÔ¡ëçz% disk./.usageGB‚1098.7493743896484% disk.nvme0n1.out‚50443.7939453125 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 5.58251953125) disk./.usagePercent‚73.73799242742002 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 496.96875 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda.in‚ 7.3671875) proc.memory.percent‚16.93868637084961 disk.sda2.out‚ 0.0390625 disk.nvme0c0n1.in‚5.3125! proc.memory.rssMB‚ 2775.234375 disk.dm-0.in‚5.3125" disk.dm-0.out‚50443.7939453125 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda.out‚ 497.0078125$ memory_percent‚97.42822647094727 network.sent‚ 2.8477558e+07 disk.nvme0n1.in‚5.3125 disk.dm-1.in‚ 7.31640625 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd3.in‚0& proc.memory.availableMB‚ 421.359375 cpu‚25.299300351397473 disk.dm-1.out‚ 496.96875 disk.rbd2.in‚0' disk.nvme0c0n1.out‚50443.7939453125 disk.sda3.in‚ 7.31640625 proc.cpu.threads‚17 network.recv‚ 3.824038e+06 disk.rbd2.out‚338.7138671875 disk.drbd1777.out‚01ŒÞ›ð*‚(: ›ùäÔ¾ü¨x gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.203' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.840600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«¶­zñ*‚(:ï ›ùäÔ¯øòz network.sent‚ 2.8586877e+07 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0) proc.memory.percent‚16.93868637084961 network.recv‚ 3.831638e+06 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda.in‚ 7.3671875 disk.sda.out‚ 497.48046875 disk.rbd1.in‚0) disk./.usagePercent‚73.73799652354057' proc.memory.availableMB‚ 421.4609375 cpu‚25.29898026498329 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625% disk./.usageGB‚1098.7494354248047 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125# memory_percent‚97.4276065826416 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 5.58251953125& disk.nvme0n1.out‚50542.98583984375 disk.dm-1.out‚ 497.44140625 proc.cpu.threads‚17 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 497.44140625 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚50542.98583984375 disk.rbd2.out‚340.2255859375! proc.memory.rssMB‚ 2775.234375 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚50542.98583984375 disk.rbd3.out‚0 disk.dm-0.in‚5.3125 disk.dm-1.in‚ 7.31640625 disk.rbd2.in‚0Ë?Xò*‚(:… ªùäÔù•¬x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô´æ¸}ó*‚(:ò ªùäÔ±ûØz% disk./.usageGB‚1098.7494354248047 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚ 7.31640625 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚50642.07177734375 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚5.3125 disk.drbd1777.out‚0 disk.dm-1.out‚ 497.97265625 network.sent‚ 2.8596176e+07 disk.rbd1.in‚0 cpu‚25.298616760773573 disk.rbd4.in‚0 disk.sda.in‚ 7.3671875 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 497.97265625 disk.rbd0.out‚0! proc.memory.rssMB‚ 2775.234375 disk.rbd1.out‚ 5.58251953125 disk.sda2.in‚0$ memory_percent‚97.42906093597412( proc.memory.availableMB‚ 421.22265625 network.recv‚ 3.842487e+06 disk.sda.out‚ 498.01171875& disk.nvme0n1.out‚50642.07177734375 disk.dm-0.in‚5.3125 disk.rbd2.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚50642.07177734375) disk./.usagePercent‚73.73799652354057 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) proc.memory.percent‚16.93868637084961 disk.rbd2.out‚340.3935546875 proc.cpu.threads‚17—à4ô*/ _wandbruntime‚10908  _runtime‚10908žœâÊõ*Ä _runtime‚10908.32356248 _step‚923 ' lr-AdamW/pg1‚1.1768094780802567e-05 ' lr-AdamW/pg2‚1.1768094780802567e-05 trainer/global_step‚11249 % _timestamp‚1.7884274375735912e+09£\Ïö*É ' lr-AdamW/pg1‚1.1768094780802567e-05 ' lr-AdamW/pg2‚1.1768094780802567e-05 trainer/global_step‚11249 % _timestamp‚1.7884274375735912e+09 _runtime‚10908.32356248 _step‚923›cÒ–4÷*/ _wandbruntime‚10908  _runtime‚10908Ç>t4¸ ø*ëO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20899066780672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚478717899998208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚467039812347904 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207819617 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚499979215604736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚446140745567232 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 500746014 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 292926397 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚446140745567232 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚10908.324515485 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚446140763570176 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 264627200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20838399562533 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 292900727 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚446140763570176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20899066780672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚467039812347904 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207819617 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚499979215604736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚446140745567232 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 500746014 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 292926397 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚446140745567232 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚10908.324515485 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚446140763570176 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262œëjÍ4ú*/ _wandbruntime‚10909  _runtime‚10909UKÍû&û*õM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚467080234206208 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197408867 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 500789310 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚478759045754368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 207837336 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 292951974 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 500763612 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20900904475136 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 121597965 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20840231962133 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 292926278 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 500763612 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500022548389888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚443327310964245 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20840455611489 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 500789310 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 197409503 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75811538 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚446179083511296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20900904475136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 207837334 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 121597963 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478759069694464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21263478695424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 207837334 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚467079987986432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 207837336 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 % _timestamp‚1.7884274384374356e+09 _step‚925 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 292926278 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 292951974 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20840455611489 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 500789310 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 197409503 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75811538 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚446179083511296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20900904475136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 207837334 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 121597963 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478759069694464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21263478695424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 207837334 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚467079987986432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 207837336 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 % _timestamp‚1.7884274384374356e+09 _step‚925 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 292926278 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 292951974 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 EDeviceStatö{v3sMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚467080234206208 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197408867 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 500789310 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚478759045754368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 207837336 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 292951974 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 500763612 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20900904475136 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 121597965 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20840231962133 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 292926278 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 500763612 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500022548389888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚443327310964245 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚448057660773376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚480776384811008 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚469048854894592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 198235417 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 208705567 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20991194121216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20990965917696 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 198234766 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 122105358 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21354780764160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 % _timestamp‚1.7884274829313753e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469048608688128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚76129408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 502912866 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 277210112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚469048854894592 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20990965917696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21355057974272 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20930033133741 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 502930464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _step‚929 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208723284 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 502956162 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 502956162 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚448095980714496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚480817554507264 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 trainer/global_step‚11299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 294207180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 % _timestamp‚1.7884274837898562e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 502930464 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20993031828992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122115715 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚10954.540450637 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20992803612160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445234308116969 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76135736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚469088784326656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20992803612160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502174474489856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚424302218934272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚448095980714496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚469089030546432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20932089182697 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚480817530567168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚448095998717440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 294232876 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198251449 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208723286 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 198252114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚448095998717440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76136399 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502174177547264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21356646980096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20993031828992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20931865533341 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884275281750698e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 209591517 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76451612 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482834869623808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21448488138240 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 199074087 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21082865054720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482834845683712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚449974539973632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21083093258240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21082865054720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_po©0@ÙŒol.allocated‚449974557976576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 122623110 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 122623108 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 209591515 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21021890354097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚471057651234816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 295462531 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚471057651234816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11349 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 209591517 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21448219316736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471057405028352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 199074722 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21021666704949 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0šÅP2ö'+ðO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 295462531 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚471057651234816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268821504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11349 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 209591517 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21448219316736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471057405028352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 199074722 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21021666704949 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚504283357762048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚449974539973632 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 505079718 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 295462531 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _step‚932 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884275281750698e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 209591517 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76451612 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482834869623808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21448488138240 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 199074087 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21082865054720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482834845683712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚449974539973632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21083093258240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21082865054720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚449974557976576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 122623110 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 122623108 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 209591515 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21021890354097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚471057651234816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800¤G%ÓÔ4ž+/ _wandbruntime‚10999  _runtime‚10999jèVû&Ÿ+õM ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21450083185664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209609234 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 209609234 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚450012877917696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482876039320064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 295488082 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122633463 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚447140823892261 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 505123014 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 505097316 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚450012877917696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482876015379968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 trainer/global_step‚11349 % _timestamp‚1.7884275290432677e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504326098565632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚447141064580977 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 295513778 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚933 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21084702749184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450012895920640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 505123014 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 295513778 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199090707 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122633465 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚426117341827072 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚10999.794078995 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21084702749184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21084930966016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 295488082 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 505097316 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397Iª, +úM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚10999.794078995 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21084702749184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21084930966016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 295488082 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 505097316 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21450083185664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209609234 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 209609234 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚450012877917696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482876039320064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 295488082 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122633463 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚447140823892261 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 505123014 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 505097316 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚450012877917696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482876015379968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 trainer/global_step‚11349 % _timestamp‚1.7884275290432677e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504326098565632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚447141064580977 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 295513778 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚933 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21084702749184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450012895920640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 505123014 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 295513778 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199090707 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122633465 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚426117341827072 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21113300276157 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 210477467 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21174992395264 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚484893330496512 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 _step‚936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚427896145222656 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427896162262016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76774008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 287695872 % _timestamp‚1.7884275737788577e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 296769103 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚76773360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 210477467 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 199914868 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 296743433 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473066201368576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 210477465 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚506435362567168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚449009686187321 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451891455179776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚451891437176832 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚484893354436608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 507246570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚449009445498813 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473066447575040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 296769103‘Ž{ñõ'¯+ïO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21113300276157 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 210477467 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21174992395264 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚484893330496512 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 _step‚936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚427896145222656 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427896162262016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76774008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 287695872 % _timestamp‚1.7884275737788577e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 296769103 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚76773360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 210477467 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 199914868 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 296743433 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473066201368576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 210477465 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚506435362567168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚449009686187321 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451891455179776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚451891437176832 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚484893354436608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 507246570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚449009445498813 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473066447575040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 296769103 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 311635968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473066447575040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 199914218 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123140860 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464¨‘ø74°+/ _wandbruntime‚11045  _runtime‚110459%Ø‹±+õM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199930738 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 296794680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 507289866 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21543602564608 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11399 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚473106623226880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210495184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 296768984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚473106377007104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451929775120896 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21176601886208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚11045.382764874 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 507289866 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451929793123840 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 210495184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 210495186 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚484934500192768 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21176830103040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21176601886208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚506478403894272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449047821044985 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 507264168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 _step‚937 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 210495186 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 296794680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884275746323018e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451929793123840 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21115132675757 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 199931363 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21115356325113 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427932447680512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76779525 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561l"‰o'²+úM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451929793123840 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 210495184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 210495186 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚484934500192768 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21176830103040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21176601886208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚506478403894272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449047821044985 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 507264168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 _step‚937 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 210495186 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 296794680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884275746323018e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451929793123840 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21115132675757 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 199931363 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21115356325113 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427932447680512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76779525 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199930738 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 296794680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 507289866 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21543602564608 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11399 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚473106623226880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210495184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 296768984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚473106377007104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451929775120896 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21176601886208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 _runtime‚11045.382764874 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 507289866 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0©eê˜4³+/ _wandbruntime‚11045  _runtime‚11045žZ;_´+Ù /loss/FRt1/assignment_loss‚3.021734204557207 2loss/SRqqt2/assignment_loss‚0.9656281385075631 % _timestamp‚1.7884275746340744e+09 &loss/total_loss‚39.807968517028755 0loss/FBt1/assignment_loss‚0.1319240060197823 1loss/SRqqt3/detection_loss‚1.7788794934749603 /loss/FBt2/detection_loss‚0.5587297876023998 /loss/FRt3/assignment_loss‚2.462152141591777 2loss/SRqqt3/assignment_loss‚1.0647598155236875 /loss/FRt1/detection_loss‚2.8631268121578075 /loss/FRt2/detection_loss‚1.5828507628212585 1loss/SRqqt2/detection_loss‚0.6732547626936738 _step‚938 0loss/FRt4/assignment_loss‚2.7169602253498177 /loss/FRt3/detection_loss‚2.3664384380630823 /loss/FBt3/detection_loss‚3.3594573102891445 epoch‚9 /loss/FBt4/detection_loss‚2.3635369300842286 trainer/global_step‚11399 0loss/FRt2/assignment_loss‚1.1162594315853525 0loss/FBt3/assignment_loss‚0.1200965981988702 .loss/FRt4/detection_loss‚2.465108742316564 1loss/SRqqt1/detection_loss‚2.2168419808149338 .loss/FBt1/detection_loss‚2.985174576441447 _runtime‚11045.383911875 2loss/SRqqt1/assignment_loss‚1.4083019939716905 1loss/SRqqt4/assignment_loss‚1.477669601328671 1loss/SRqqt4/detection_loss‚1.7440934235659928 1loss/FBt2/assignment_loss‚0.13664775234863724 1loss/FBt4/assignment_loss‚0.22834158772020602×–wødµ+Þ /loss/FRt1/assignment_loss‚3.021734204557207 2loss/SRqqt2/assignment_loss‚0.9656281385075631 % _timestamp‚1.7884275746340744e+09 &loss/total_loss‚39.807968517028755 0loss/FBt1/assignment_loss‚0.1319240060197823 1loss/SRqqt3/detection_loss‚1.7788794934749603 /loss/FBt2/detection_loss‚0.5587297876023998 /loss/FRt3/assignment_loss‚2.462152141591777 2loss/SRqqt3/assignment_loss‚1.0647598155236875 /loss/FRt1/detection_loss‚2.8631268121578075 /loss/FRt2/detection_loss‚1.5828507628212585 1loss/SRqqt2/detection_loss‚0.6732547626936738 _step‚938 0loss/FRt4/assignment_loss‚2.7169602253498177 /loss/FRt3/detection_loss‚2.3664384380630823 /loss/FBt3/detection_loss‚3.3594573102891445 epoch‚9 /loss/FBt4/detection_loss‚2.3635369300842286 trainer/global_step‚11399 0loss/FRt2/assignment_loss‚1.1162594315853525 0loss/FBt3/assignment_loss‚0.1200965981988702 .loss/FRt4/detection_loss‚2.465108742316564 1loss/SRqqt1/detection_loss‚2.2168419808149338 .loss/FBt1/detection_loss‚2.985174576441447 _runtime‚11045.383911875 2loss/SRqqt1/assignment_loss‚1.4083019939716905 1loss/SRqqt4/assignment_loss‚1.477669601328671 1loss/SRqqt4/detection_loss‚1.7440934235659928 1loss/FBt2/assignment_loss‚0.13664775234863724 1loss/FBt4/assignment_loss‚0.22834158772020602ª‰~ˆÆŽ¶+‚(:ƒ ÀúäÔûÞ½x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e8™q·+‚(:æ ÀúäÔå…¦{ disk.nvme0c0n1.in‚5.3125) disk./.usagePercent‚73.73791920926523% disk./.usageGB‚1098.7482833862305% disk.nvme0n1.out‚51440.6376953125 disk.sda2.in‚0 disk.dm-1.in‚ 7.31640625 disk.rbd4.out‚0 disk.sda3.in‚ 7.31640625 cpu‚25.295203437225727 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚51440.6376953125 disk.sda1.out‚0 disk.sda.out‚ 506.8515625 disk.sda3.out‚506.8125( proc.memory.availableMB‚ 421.31640625 disk.dm-1.out‚506.8125 network.sent‚ 2.8966103e+07" disk.dm-0.out‚51440.6376953125 disk.rbd2.out‚350.2998046875 disk.drbd1777.in‚ 1.03515625) proc.memory.percent‚16.93868637084961 disk.sda1.in‚ 0.05078125$ memory_percent‚97.42848873138428 network.recv‚ 3.898142e+06 disk.rbd1.out‚ 5.58251953125 disk.dm-0.in‚5.3125 disk.rbd3.in‚0 disk.sda.in‚ 7.3671875 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd3.out‚0! proc.memory.rssMB‚ 2775.234375 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0Ûúø+‚(:„ ÏúäÔ¨ü¡x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.932' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3864 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û% Ä|¹+‚(:ñ ÏúäÔÉîîz disk.dm-0.in‚5.3125 disk.sda1.out‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚ 7.31640625 disk.dm-1.in‚ 7.31640625 disk.drbd1777.out‚0 disk.sda.out‚ 507.64453125' proc.memory.availableMB‚ 421.2734375 disk.rbd1.out‚ 5.58251953125 disk.nvme0n1.in‚5.3125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.sda3.out‚ 507.60546875 disk.rbd1.in‚0 disk.rbd3.in‚0$ memory_percent‚97.42875099182129) proc.memory.percent‚16.93868637084961 network.sent‚ 2.9075656e+07& disk.nvme0n1.out‚51441.43017578125 disk.sda2.out‚ 0.0390625 disk.rbd2.out‚351.8115234375( disk.nvme0c0n1.out‚51441.43017578125% disk./.usageGB‚1098.7482833862305 disk.sda2.in‚0 disk.dm-1.out‚ 507.60546875 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.3671875 network.recv‚ 3.905686e+06) disk./.usagePercent‚73.73791920926523 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 cpu‚25.294874031093926# disk.dm-0.out‚51441.43017578125! proc.memory.rssMB‚ 2775.234375 proc.cpu.threads‚177áBº+‚(:… Þúäԩ¨x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.896' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë:{»+‚(:ð ÞúäÔ¹³Ýz disk.nvme0c0n1.in‚5.3125$ memory_percent‚97.42872714996338 cpu‚25.2945441132874 proc.cpu.threads‚17 disk.rbd1.out‚ 5.58251953125 disk.sda2.in‚0) disk./.usagePercent‚73.73791920926523% disk./.usageGB‚1098.7482833862305& disk.nvme0n1.out‚51540.60205078125 disk.rbd0.in‚0( proc.memory.availableMB‚ 421.27734375 disk.sda3.in‚ 7.31640625 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 507.99609375 disk.rbd2.in‚0 disk.rbd2.out‚351.9130859375 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.nvme0n1.in‚5.3125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.in‚ 7.3671875 disk.sda.out‚ 508.03515625( disk.nvme0c0n1.out‚51540.60205078125) proc.memory.percent‚16.93868637084961 disk.sda3.out‚ 507.99609375 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0! proc.memory.rssMB‚ 2775.234375 network.sent‚ 2.9081766e+07 network.recv‚ 3.909815e+06 disk.rbd3.in‚0 disk.drbd1777.in‚ 1.03515625# disk.dm-0.out‚51540.60205078125 disk.dm-1.in‚ 7.31640625 disk.rbd4.in‚0 disk.rbd4.out‚0h†4¼+/ _wandbruntime‚11089  _runtime‚11089>j°(ǽ+Á _runtime‚11089.887176831 _step‚939 % lr-AdamW/pg1‚6.85054523459563e-06 % lr-AdamW/pg2‚6.85054523459563e-06 trainer/global_step‚11449 % _timestamp‚1.7884276191372344e+09ù+{̾+Æ trainer/global_step‚11449 % _timestamp‚1.7884276191372344e+09 _runtime‚11089.887176831 _step‚939 % lr-AdamW/pg1‚6.85054523459563e-06 % lr-AdamW/pg2‚6.85054523459563e-06«œ´–4¿+/ _wandbruntime‚11089  _runtime‚11089³ç·À+êO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚508587044544000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 trainer/global_step‚11449 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475074997708800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 298024335 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 509413422 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚475075243915264 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 211363417 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _step‚940 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚486951839249408 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 211363415 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453808352382976 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21266891532288 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453808334380032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚508587341499904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21204933847365 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 200752344 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 298050005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚450916442651329 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚11089.888128263 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0,êwõ'Á+ïO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 211363415 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453808352382976 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21266891532288 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453808334380032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚508587341499904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21204933847365 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 200752344 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 298050005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚450916442651329 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚11089.888128263 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚508587044544000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 trainer/global_step‚11449 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475074997708800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 298024335 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 509413422 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚475075243915264 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 211363417 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 _step‚940 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚486951839249408¬0ÀÚ4Â+/ _wandbruntime‚11090  _runtime‚11090) ¸´û&Ã+õM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21206766246965 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚475115419567104 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 123668963 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 211381134 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚508630081202176 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 509456718 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚429747587612672 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21268729240064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21637096196608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚486992985005568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21268729240064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 298075582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚429747570573312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21206989896321 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 _runtime‚11090.740378042 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 298075582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚450954336820277 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 211381136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 298049886 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚77100776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚450954577508993 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚453846672324096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21268501023232 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 298049886 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 200769741 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚453846672324096 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 200769099 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77100136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11449 % _timestamp‚1.7884276199898446e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚453846690327040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 509456718 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 509431020 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚475115419567104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 123668965 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21268729240064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21637096196608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚486992985005568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21268729240064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 298075582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚429747570573312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21206989896321 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 _runtime‚11090.740378042 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 298075582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚450954336820277 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 211381136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 298049886 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚77100776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚450954577508993 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚453846672324096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21268501023232 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 298049886 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 200769741 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚453846672324096 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 200769099 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77100136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11449 % _timestamp‚1.7884276199898446e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚453846690327040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 509456718 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 509431020 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚475115419567104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 123668965 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21206766246965 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚475115419567104 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 123668963 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚rèËK31397 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 211381134 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚508630081202176 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 509456718 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚429747587612672­ ñš4Å+/ _wandbruntime‚11090  _runtime‚11090)Z:bÆ+Ü 0loss/FRt2/assignment_loss‚0.9979777283138699 1loss/FBt2/assignment_loss‚0.16601377577545096 /loss/FRt3/detection_loss‚2.6150090475049286 1loss/SRqqt2/detection_loss‚0.6579319255378766 -loss/FBt1/detection_loss‚2.14335815442933 2loss/SRqqt1/assignment_loss‚1.6445186244029748 2loss/FBt3/assignment_loss‚0.023693817388266324 0loss/FRt1/assignment_loss‚2.5928616251634513 /loss/FBt3/detection_loss‚2.7335466861724855 % _timestamp‚1.7884276199917555e+09 _runtime‚11090.741586679 /loss/FRt1/detection_loss‚2.4207907450263915 0loss/SRqqt1/detection_loss‚2.336025297641754 1loss/SRqqt4/detection_loss‚1.7131719823394502 %loss/total_loss‚37.89333361962319 0loss/FRt4/assignment_loss‚3.1583743890126548 2loss/SRqqt3/assignment_loss‚1.3000695725359643 2loss/SRqqt4/assignment_loss‚1.4882033066253881 /loss/FRt4/detection_loss‚2.2956162518880725 0loss/SRqqt3/detection_loss‚1.739674503604571 /loss/FRt2/detection_loss‚1.4513331360287136 /loss/FBt2/detection_loss‚0.6689747222495498 epoch‚9 0loss/FRt3/assignment_loss‚2.1917135814825692 1loss/FBt1/assignment_loss‚0.38552344608635597 /loss/FBt4/detection_loss‚2.3024244755506516 2loss/SRqqt2/assignment_loss‚0.7045170333331976 1loss/FBt4/assignment_loss‚0.16200979152927175 trainer/global_step‚11449 _step‚942›ÏÀgÇ+á 1loss/SRqqt4/detection_loss‚1.7131719823394502 %loss/total_loss‚37.89333361962319 0loss/FRt4/assignment_loss‚3.1583743890126548 2loss/SRqqt3/assignment_loss‚1.3000695725359643 2loss/SRqqt4/assignment_loss‚1.4882033066253881 /loss/FRt4/detection_loss‚2.2956162518880725 0loss/SRqqt3/detection_loss‚1.739674503604571 /loss/FRt2/detection_loss‚1.4513331360287136 /loss/FBt2/detection_loss‚0.6689747222495498 epoch‚9 0loss/FRt3/assignment_loss‚2.1917135814825692 1loss/FBt1/assignment_loss‚0.38552344608635597 /loss/FBt4/detection_loss‚2.3024244755506516 2loss/SRqqt2/assignment_loss‚0.7045170333331976 1loss/FBt4/assignment_loss‚0.16200979152927175 trainer/global_step‚11449 _step‚942 0loss/FRt2/assignment_loss‚0.9979777283138699 1loss/FBt2/assignment_loss‚0.16601377577545096 /loss/FRt3/detection_loss‚2.6150090475049286 1loss/SRqqt2/detection_loss‚0.6579319255378766 -loss/FBt1/detection_loss‚2.14335815442933 2loss/SRqqt1/assignment_loss‚1.6445186244029748 2loss/FBt3/assignment_loss‚0.023693817388266324 0loss/FRt1/assignment_loss‚2.5928616251634513 /loss/FBt3/detection_loss‚2.7335466861724855 % _timestamp‚1.7884276199917555e+09 _runtime‚11090.741586679 /loss/FRt1/detection_loss‚2.4207907450263915 0loss/SRqqt1/detection_loss‚2.336025297641754®€tˆšÈ+‚(: íúäÔ©’x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.811799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uëoixÉ+‚(:í íúäÔÁãÈ{ disk.rbd4.out‚0% disk./.usageGB‚1098.7482833862305 disk.sda.out‚ 508.640625 disk.dm-0.in‚5.3125# disk.dm-0.out‚51640.04638671875 disk.sda.in‚ 7.3671875 disk.sda3.in‚ 7.31640625 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.dm-1.in‚ 7.31640625 disk.rbd2.in‚0 disk.rbd2.out‚353.4248046875 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda2.out‚ 0.0390625 network.sent‚ 2.909177e+07 disk.nvme0c0n1.in‚5.3125 disk.rbd3.out‚0& disk.nvme0n1.out‚51640.04638671875( proc.memory.availableMB‚ 420.74609375 disk.sda2.in‚0 network.recv‚ 3.921307e+06) disk./.usagePercent‚73.73791920926523 disk.rbd4.in‚0 disk.sda3.out‚ 508.6015625 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625( disk.nvme0c0n1.out‚51640.04638671875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 5.58251953125 disk.dm-1.out‚ 508.6015625 disk.rbd0.out‚0 cpu‚25.294232979949157 disk.drbd1777.out‚0$ memory_percent‚97.43196964263916! proc.memory.rssMB‚ 2775.234375) proc.memory.percent‚16.93868637084961äl=í›Ê+‚(: üúäԼȂx gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.969' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.993799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A™èFuË+‚(:ê üúäÔ÷Úª{ disk.drbd1777.out‚0( disk.nvme0c0n1.out‚51739.26611328125 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 509.09375 disk.nvme0n1.in‚5.3125( proc.memory.availableMB‚ 421.22265625 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚51739.26611328125 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 2.9199826e+07$ disk./.usageGB‚1098.748275756836 disk.sda.in‚ 7.3671875 disk.sda.out‚ 509.1328125 disk.dm-0.in‚5.3125! proc.memory.rssMB‚ 2775.234375) disk./.usagePercent‚73.73791869725017 disk.rbd1.in‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125# disk.dm-0.out‚51739.26611328125 disk.dm-1.out‚ 509.09375 disk.rbd2.in‚0 disk.rbd1.out‚ 5.58251953125 disk.rbd2.out‚353.5927734375) proc.memory.percent‚16.93868637084961 network.recv‚ 3.926985e+06 disk.dm-1.in‚ 7.31640625$ memory_percent‚97.42906093597412 cpu‚25.293882852908325 disk.drbd1777.in‚ 1.03515625žV²ˆÌ+‚(:… ‹ûäÔ“¦x gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.598' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c¹Ë‡|Í+‚(:ñ ‹ûäÔ¹¤ðz disk.rbd0.in‚0) disk./.usagePercent‚73.73793277766454 disk.sda1.in‚ 0.05078125 disk.sda3.in‚ 7.31640625 disk.sda2.in‚0) proc.memory.percent‚16.93868637084961% disk./.usageGB‚1098.7484855651855 disk.sda1.out‚0 disk.dm-0.in‚5.3125 disk.rbd2.out‚355.1162109375! proc.memory.rssMB‚ 2775.234375 cpu‚25.29357944501325 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚97.42801189422607 network.sent‚ 2.9212713e+07 disk.sda.out‚ 509.68359375 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda3.out‚ 509.64453125# disk.dm-0.out‚51740.33837890625 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚ 7.31640625( disk.nvme0c0n1.out‚51740.33837890625 disk.rbd1.out‚ 5.58251953125& disk.nvme0n1.out‚51740.33837890625 disk.dm-1.out‚ 509.64453125 network.recv‚ 3.945395e+06 disk.sda.in‚ 7.3671875 disk.nvme0n1.in‚5.3125 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125( proc.memory.availableMB‚ 421.39453125 proc.cpu.threads‚17`u…ò4Î+/ _wandbruntime‚11135  _runtime‚11135X•zÉÏ+à trainer/global_step‚11499 % _timestamp‚1.7884276645811727e+09 _runtime‚11135.331119271 _step‚943 & lr-AdamW/pg1‚5.825993317534767e-06 & lr-AdamW/pg2‚5.825993317534767e-06æGUÎÐ+È _runtime‚11135.331119271 _step‚943 & lr-AdamW/pg1‚5.825993317534767e-06 & lr-AdamW/pg2‚5.825993317534767e-06 trainer/global_step‚11499 % _timestamp‚1.7884276645811727e+09¯|'4Ñ+/ _wandbruntime‚11135  _runtime‚11135TSü²ð'Ò+êO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚489010300122112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚477084040255488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 511554602 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚455725249586176 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚510738930351616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚431526391008256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 201593009 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚11499 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚452823199115337 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431526408047616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚455725231583232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚489010324062208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124176358 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21296567418573 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚455725249586176 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 212249365 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 212249367 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21728630229504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21296791067721 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77416649 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21358562465792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 299330907 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 $ _timestamp‚1.788427664581932e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 299305237 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124176360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _runtime‚11135.332047058 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚477083794049024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 212249367 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477083794049024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 299305237 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 201592330 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21728903245312 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _step‚944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 511580274 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096¹õ'Ó+ïO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚455725231583232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚489010324062208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124176358 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21296567418573 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚455725249586176 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 212249365 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 212249367 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21728630229504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21296791067721 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77416649 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21358562465792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 299330907 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 $ _timestamp‚1.788427664581932e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 299305237 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124176360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 _runtime‚11135.332047058 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚477083794049024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 212249367 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477083794049024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 299305237 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 201592330 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21728903245312 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _step‚944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 511580274 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚489010300122112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚477084040255488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 511554602 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚455725249586176 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚510738930351616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚431526391008256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 201593009 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 trainer/global_step‚11499 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚452823199115337 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431526408047616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428°xÅ(­4Ô+/ _wandbruntime‚11136  _runtime‚11136J؉Ï!Õ+õM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚510781962182144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 299356484 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 201609548 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21360400160256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21360400160256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 trainer/global_step‚11499 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 201608914 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477124215907328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚489051493758464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21298399818173 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477124215907328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21360628377088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431562693466112 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚452861333973001 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21298623467529 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 511597872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21730492363776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _runtime‚11136.208339055 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 511623570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚77422833 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚510782261221888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 % _timestamp‚1.7884276654577518e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 212267086 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124186713 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77422201 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚452861093284285 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 299356484 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 212267086 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚489051469818368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431562710505472 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 511623570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744ÍJ¢ü'Ö+úM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21360400160256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21360400160256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 trainer/global_step‚11499 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 201608914 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477124215907328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚489051493758464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21298399818173 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477124215907328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21360628377088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431562693466112 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚452861333973001 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21298623467529 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 511597872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21730492363776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 _runtime‚11136.208339055 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 511623570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚77422833 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚510782261221888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 % _timestamp‚1.7884276654577518e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 212267086 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124186713 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77422201 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚452861093284285 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 299356484 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 212267086 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚489051469818368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431562710505472 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 511623570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚510781962182144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 299356484 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 201609548 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360±Œ×(4×+/ _wandbruntime‚11136  _runtime‚11136)®™®ZØ+Ô 0loss/FRt2/assignment_loss‚1.4121641476045956 /loss/FRt3/assignment_loss‚2.894956547589529 2loss/FBt1/assignment_loss‚0.008158780484547736 0loss/FBt3/assignment_loss‚0.2541718321048062 /loss/FBt2/detection_loss‚0.4928577494060317 .loss/FRt1/detection_loss‚2.582087977936393 1loss/SRqqt2/detection_loss‚0.8971836600256594 /loss/FBt4/assignment_loss‚0.280631756875664 0loss/SRqqt3/detection_loss‚2.008680880717609 0loss/SRqqt4/detection_loss‚1.983880838873435 /loss/FBt3/detection_loss‚2.3554111664945427 1loss/SRqqt1/detection_loss‚2.5647905748337507 _step‚946 .loss/FRt2/detection_loss‚2.064785740592263 2loss/SRqqt2/assignment_loss‚0.9331020708398826 /loss/FBt1/detection_loss‚3.1431953396115984 .loss/FBt4/detection_loss‚2.516945117049747 epoch‚9 % _timestamp‚1.7884276654595923e+09 0loss/FRt1/assignment_loss‚3.2008704511742843 1loss/SRqqt1/assignment_loss‚1.582957522070501 2loss/SRqqt3/assignment_loss‚1.5044027559621178 /loss/FRt4/detection_loss‚2.2514824794264054 0loss/FRt4/assignment_loss‚2.5934648367823385 2loss/SRqqt4/assignment_loss‚1.4782828686530798 -loss/FRt3/detection_loss‚2.43011008699735 %loss/total_loss‚41.50200534356425 1loss/FBt2/assignment_loss‚0.06743016145812229 _runtime‚11136.209442544 trainer/global_step‚11499ôç*Ü_Ù+Ù /loss/FBt4/assignment_loss‚0.280631756875664 0loss/SRqqt3/detection_loss‚2.008680880717609 0loss/SRqqt4/detection_loss‚1.983880838873435 /loss/FBt3/detection_loss‚2.3554111664945427 1loss/SRqqt1/detection_loss‚2.5647905748337507 _step‚946 .loss/FRt2/detection_loss‚2.064785740592263 2loss/SRqqt2/assignment_loss‚0.9331020708398826 /loss/FBt1/detection_loss‚3.1431953396115984 .loss/FBt4/detection_loss‚2.516945117049747 epoch‚9 % _timestamp‚1.7884276654595923e+09 0loss/FRt1/assignment_loss‚3.2008704511742843 1loss/SRqqt1/assignment_loss‚1.582957522070501 2loss/SRqqt3/assignment_loss‚1.5044027559621178 /loss/FRt4/detection_loss‚2.2514824794264054 0loss/FRt4/assignment_loss‚2.5934648367823385 2loss/SRqqt4/assignment_loss‚1.4782828686530798 -loss/FRt3/detection_loss‚2.43011008699735 %loss/total_loss‚41.50200534356425 1loss/FBt2/assignment_loss‚0.06743016145812229 _runtime‚11136.209442544 trainer/global_step‚11499 0loss/FRt2/assignment_loss‚1.4121641476045956 /loss/FRt3/assignment_loss‚2.894956547589529 2loss/FBt1/assignment_loss‚0.008158780484547736 0loss/FBt3/assignment_loss‚0.2541718321048062 /loss/FBt2/detection_loss‚0.4928577494060317 .loss/FRt1/detection_loss‚2.582087977936393 1loss/SRqqt2/detection_loss‚0.8971836600256594²ÐØîÚ+‚(:… šûäÔƪ¦y gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E˜p>|Û+‚(:ñ šûäÔ—Êþz) disk./.usagePercent‚73.73793277766454 disk.rbd4.out‚0 disk.sda.out‚ 510.19140625& disk.nvme0n1.out‚51839.54443359375( proc.memory.availableMB‚ 421.02734375 disk.dm-1.out‚ 510.15234375 disk.rbd0.out‚0 disk.rbd3.in‚0) proc.memory.percent‚16.93868637084961 disk.sda2.in‚0 disk.rbd2.out‚355.2177734375 disk.rbd3.out‚0 disk.sda3.out‚ 510.15234375 disk.rbd1.out‚ 5.58251953125 disk.nvme0n1.in‚5.3125# disk.dm-0.out‚51839.54443359375$ memory_percent‚97.43025302886963! proc.memory.rssMB‚ 2775.234375 network.recv‚ 3.956275e+06 disk.rbd1.in‚0% disk./.usageGB‚1098.7484855651855 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚51839.54443359375 disk.sda1.out‚0 disk.sda.in‚ 7.3671875 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚ 7.31640625 disk.drbd1777.in‚ 1.03515625 disk.sda3.in‚ 7.31640625 disk.dm-0.in‚5.3125 cpu‚25.29322947666287 proc.cpu.threads‚17 network.sent‚ 2.9222111e+07 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125éƽÜÜ+‚(:… ©ûäÔ’Á¬x gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.654' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßv…vÝ+‚(:ë ©ûäÔ™´Ùz disk.nvme0n1.in‚5.3125$ memory_percent‚97.43051528930664! proc.memory.rssMB‚ 2775.234375) proc.memory.percent‚16.93868637084961 cpu‚25.292860048429343 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 510.703125 disk.dm-0.in‚5.3125& proc.memory.availableMB‚ 420.984375 proc.cpu.threads‚17 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.3671875 disk.sda2.in‚0 disk.rbd0.in‚0 network.recv‚ 3.970309e+06) disk./.usagePercent‚73.73793277766454 disk.rbd3.out‚0 disk.rbd2.out‚356.7294921875 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 510.7421875 disk.rbd1.in‚0 disk.rbd1.out‚ 5.58251953125 network.sent‚ 2.9334521e+07 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.dm-1.in‚ 7.31640625 disk.rbd2.in‚0 disk.dm-1.out‚ 510.703125 disk.nvme0c0n1.in‚5.3125( disk.nvme0c0n1.out‚51938.77978515625& disk.nvme0n1.out‚51938.77978515625# disk.dm-0.out‚51938.77978515625 disk.sda2.out‚ 0.0390625% disk./.usageGB‚1098.7484855651855 disk.drbd1777.out‚0 disk.sda3.in‚ 7.31640625• ¸‰šÞ+‚(: ¸ûäÔþ¯¡x gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.46' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.491999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?#Mqß+‚(:æ ¸ûäÔ³•Ë{! proc.memory.rssMB‚ 2775.234375 cpu‚25.292530655688303 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625) proc.memory.percent‚16.93868637084961 network.recv‚ 3.973588e+06 disk.rbd1.out‚ 5.58251953125 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda1.out‚0' proc.memory.availableMB‚ 420.8828125) disk./.usagePercent‚73.73792714549879 disk.dm-0.in‚5.3125" disk.dm-0.out‚52037.9560546875 disk.dm-1.out‚ 511.1171875 disk.rbd4.out‚0# memory_percent‚97.4311351776123% disk./.usageGB‚1098.7484016418457 disk.sda.in‚ 7.3671875 disk.sda3.out‚ 511.1171875 disk.rbd2.out‚ 358.48046875 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 511.15625 disk.sda3.in‚ 7.31640625 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.nvme0c0n1.in‚5.3125' disk.nvme0c0n1.out‚52037.9560546875 disk.rbd3.in‚0% disk.nvme0n1.out‚52037.9560546875 disk.dm-1.in‚ 7.31640625 disk.rbd2.in‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 network.sent‚ 2.9339851e+07 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0Ã÷¼q4à+/ _wandbruntime‚11180  _runtime‚11180ÄŒ¦Éá+à trainer/global_step‚11549 % _timestamp‚1.7884277102418795e+09 _runtime‚11180.991802911 _step‚947 & lr-AdamW/pg1‚4.883828276450991e-06 & lr-AdamW/pg2‚4.883828276450991e-06r(·èÎâ+È % _timestamp‚1.7884277102418795e+09 _runtime‚11180.991802911 _step‚947 & lr-AdamW/pg1‚4.883828276450991e-06 & lr-AdamW/pg2‚4.883828276450991e-06 trainer/global_step‚11549³7åŽô4ã+/ _wandbruntime‚11180  _runtime‚11180Ó¸œ­ä+ëO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚479092590389248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚457642146789376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77738372 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚491068808875008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21450461602816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21388424638929 % _timestamp‚1.7884277102426271e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21450689806336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 202431871 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 300586139 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 202432482 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚491068784934912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚479092836595712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124694108 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 213135315 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚948 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77737763 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21822262418944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚454729714890837 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 300586139 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 213135317 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚433341530940416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚479092590389248 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124694110 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21821995694592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 513721454 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21450689806336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 213135317 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 300611809 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433341513901056 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 trainer/global_step‚11549 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚457642146789376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚454729955579345 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚457642128786432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚512891071293952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚512890780629504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21450461602816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚479092836595712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 513747126 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚948 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77737763 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21822262418944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚454729714890837 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 300586139 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 213135317 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚433341530940416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚479092590389248 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124694110 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266724352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21821995694592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 513721454 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21450689806336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 213135317 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 300611809 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433341513901056 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 trainer/global_step‚11549 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚457642146789376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚454729955579345 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚457642128786432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚512891071293952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚512890780629504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 290664448 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21450461602816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚479092836595712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 513747126 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 513790422 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚512933817733632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21823863102464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433377816358912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 300637386 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21452299297280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚457680484733440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 300611690 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚949 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚479133012247552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21452527514112 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚479132766027776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 202449092 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚454767849748293 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 213153036 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 202448439 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 213153034 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21452299297280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21824134007808 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21390257038737 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚457680484733440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77743976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚457680466730496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 213153036 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚512934112579072 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 513790422 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 300637386 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124704463 $ _timestamp‚1.788427711097095e+09 _runtime‚11181.847729235 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 513764724 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 300611690 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚479133012247552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚457680466730496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 513764724 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚454768090437009 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚479133012247552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚457680466730496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 513764724 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚454768090437009 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 513790422 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚512933817733632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21823863102464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433377816358912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 300637386 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21452299297280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚457680484733440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 300611690 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚949 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚479133012247552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21452527514112 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚479132766027776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 202449092 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚454767849748293 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 213153036 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 202448439 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 213153034 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21452299297280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21824134007808 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21390257038737 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚457680484733440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77743976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚457680466730496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 213153036 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚512934112579072 WIDeviceStatsMonitor.on_trîhÃXŠain_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 513790422 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 300637386 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124704463 $ _timestamp‚1.788427711097095e+09 _runtime‚11181.847729235 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 513764724 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262µ…4´4é+/ _wandbruntime‚11181  _runtime‚11181ELQª^ê+Ø 1loss/SRqqt2/assignment_loss‚0.682389156103543 1loss/FBt1/assignment_loss‚0.23470541292850353 1loss/FBt3/assignment_loss‚0.20239318716630805 0loss/FRt1/assignment_loss‚2.9966020504633586 1loss/FBt2/assignment_loss‚0.09847838073379643 /loss/FRt4/detection_loss‚2.7589532918221247 1loss/SRqqt1/detection_loss‚2.7732327145834765 0loss/SRqqt2/detection_loss‚0.813065018005124 epoch‚9 _step‚950 0loss/FRt4/assignment_loss‚2.8051277286297567 2loss/SRqqt1/assignment_loss‚1.4946229675163825 2loss/SRqqt3/assignment_loss‚1.6113820692487792 0loss/SRqqt4/detection_loss‚2.003801276690738 % _timestamp‚1.7884277110990686e+09 0loss/FRt3/assignment_loss‚2.9114400034859065 .loss/FRt1/detection_loss‚2.864736333737771 /loss/FBt3/detection_loss‚3.3596457690000534 %loss/total_loss‚40.87880601257887 trainer/global_step‚11549 .loss/FRt3/detection_loss‚2.404978328517505 _runtime‚11181.848895211 /loss/FRt2/assignment_loss‚1.201473519206047 2loss/FBt4/assignment_loss‚0.036487379311842635 .loss/FRt2/detection_loss‚1.374962957176779 0loss/SRqqt3/detection_loss‚1.737330789367358 /loss/FBt2/detection_loss‚0.6733100436729927 /loss/FBt4/detection_loss‚2.1173883742756314 2loss/SRqqt4/assignment_loss‚1.0797643705899285 /loss/FBt1/detection_loss‚2.6425348903451646¢æcë+Ý 1loss/FBt2/assignment_loss‚0.09847838073379643 /loss/FRt4/detection_loss‚2.7589532918221247 1loss/SRqqt1/detection_loss‚2.7732327145834765 0loss/SRqqt2/detection_loss‚0.813065018005124 epoch‚9 _step‚950 0loss/FRt4/assignment_loss‚2.8051277286297567 2loss/SRqqt1/assignment_loss‚1.4946229675163825 2loss/SRqqt3/assignment_loss‚1.6113820692487792 0loss/SRqqt4/detection_loss‚2.003801276690738 % _timestamp‚1.7884277110990686e+09 0loss/FRt3/assignment_loss‚2.9114400034859065 .loss/FRt1/detection_loss‚2.864736333737771 /loss/FBt3/detection_loss‚3.3596457690000534 %loss/total_loss‚40.87880601257887 trainer/global_step‚11549 .loss/FRt3/detection_loss‚2.404978328517505 _runtime‚11181.848895211 /loss/FRt2/assignment_loss‚1.201473519206047 2loss/FBt4/assignment_loss‚0.036487379311842635 .loss/FRt2/detection_loss‚1.374962957176779 0loss/SRqqt3/detection_loss‚1.737330789367358 /loss/FBt2/detection_loss‚0.6733100436729927 /loss/FBt4/detection_loss‚2.1173883742756314 2loss/SRqqt4/assignment_loss‚1.0797643705899285 /loss/FBt1/detection_loss‚2.6425348903451646 1loss/SRqqt2/assignment_loss‚0.682389156103543 1loss/FBt1/assignment_loss‚0.23470541292850353 1loss/FBt3/assignment_loss‚0.20239318716630805 0loss/FRt1/assignment_loss‚2.9966020504633586¶ɈAŸŽì+‚(:ƒ ÇûäÔÛ¨Ÿx gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚96.555' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.311 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™C±¶qí+‚(:æ ÇûäÔüÏÙz network.recv‚ 3.978165e+06 disk.rbd3.in‚0' disk.nvme0c0n1.out‚52137.2744140625 disk.rbd1.in‚0% disk.nvme0n1.out‚52137.2744140625 disk.sda2.in‚0 disk.dm-1.out‚ 511.69921875 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625# memory_percent‚97.4325180053711 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 511.69921875 disk.rbd2.in‚0! proc.memory.rssMB‚ 2775.234375) proc.memory.percent‚16.93868637084961 proc.cpu.threads‚17" disk.dm-0.out‚52137.2744140625% disk./.usageGB‚1098.7486457824707 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.rbd1.out‚ 5.58251953125 disk.dm-1.in‚ 7.31640625 disk.rbd2.out‚ 358.5078125 disk.sda1.in‚ 0.05078125 disk.nvme0c0n1.in‚5.3125 network.sent‚ 2.934598e+07) disk./.usagePercent‚73.73794352998098 disk.dm-0.in‚5.3125 disk.rbd0.in‚0 disk.sda.in‚ 7.3671875 disk.drbd1777.out‚0 disk.rbd4.out‚0 disk.sda1.out‚0 disk.rbd3.out‚0% proc.memory.availableMB‚ 420.65625 disk.sda.out‚ 511.73828125 disk.rbd4.in‚0 cpu‚25.29221046660446[î+‚(:… ÖûäÔÀ¸—x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.391' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.6782 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂAF½yï+‚(:î ÖûäÔæÔê{ proc.cpu.threads‚17" disk.dm-0.out‚52137.6025390625 disk.dm-1.out‚ 512.01953125 disk.rbd4.in‚0! proc.memory.rssMB‚ 2775.234375 disk.sda1.out‚0 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 network.recv‚ 3.984264e+06 disk.sda2.out‚ 0.0390625$ memory_percent‚97.43218421936035) proc.memory.percent‚16.93868637084961 disk.sda3.out‚ 512.01953125 disk.rbd1.out‚ 5.58251953125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0' proc.memory.availableMB‚ 420.7109375% disk./.usageGB‚1098.7486686706543 disk.rbd3.out‚0 disk.sda3.in‚ 7.31640625 disk.dm-1.in‚ 7.31640625 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125% disk.nvme0n1.out‚52137.6025390625! disk.rbd2.out‚360.16845703125 network.sent‚ 2.9454696e+07' disk.nvme0c0n1.out‚52137.6025390625 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 512.05859375 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚73.73794506602619 disk.rbd3.in‚0 disk.sda.in‚ 7.3671875 cpu‚25.29188168330779üàdŽð+‚(:ƒ åûäԉǑx gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.37' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡˜Åšwñ+‚(:ì åûäÔ¾ê{ disk.drbd1777.out‚0 network.recv‚ 3.98928e+06 disk.rbd3.in‚0 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 512.609375 disk.rbd1.out‚ 5.58251953125# disk.dm-0.out‚52237.55224609375 disk.dm-1.in‚ 7.31640625 disk.sda1.out‚0 disk.rbd4.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚5.3125 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625( proc.memory.availableMB‚ 420.33203125! proc.memory.rssMB‚ 2775.234375 disk.sda2.out‚ 0.0390625( disk.nvme0c0n1.out‚52237.55224609375 disk.sda1.in‚ 0.05078125 disk.rbd2.in‚0 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚52237.55224609375) disk./.usagePercent‚73.73791716120496% disk./.usageGB‚1098.7482528686523 disk.sda.in‚ 7.3671875$ memory_percent‚97.43449687957764) proc.memory.percent‚16.93868637084961 disk.rbd3.out‚0 disk.sda.out‚ 512.6484375 disk.sda2.in‚0 disk.dm-1.out‚ 512.609375! disk.rbd2.out‚360.19189453125 disk.rbd4.out‚0 cpu‚25.29155909793707 disk.nvme0c0n1.in‚5.3125 network.sent‚ 2.9461096e+07 disk.rbd1.in‚0 disk.rbd0.in‚0Â7E-4ò+/ _wandbruntime‚11226  _runtime‚11226`ùÂ8Éó+à _step‚951 & lr-AdamW/pg1‚4.024246457126805e-06 & lr-AdamW/pg2‚4.024246457126805e-06 trainer/global_step‚11599 % _timestamp‚1.7884277554271557e+09 _runtime‚11226.177118623ø +ÚÎô+È % _timestamp‚1.7884277554271557e+09 _runtime‚11226.177118623 _step‚951 & lr-AdamW/pg1‚4.024246457126805e-06 & lr-AdamW/pg2‚4.024246457126805e-06 trainer/global_step‚11599·øçn64õ+/ _wandbruntime‚11226  _runtime‚11226Tº;}ñ'ö+ëO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 301892711 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚78061592 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚481101632935936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚459559025989632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 301892711 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21915680625664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 203273452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456636712043353 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚481101386729472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚435156653833216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 515913978 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21479834560989 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚515042974313472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 515888306 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚481101386729472 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 214021267 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125211858 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456636471354845 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 203272774 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 214021265 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21915405512704 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚11226.178082128¶ äŽö'÷+ðO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚481101386729472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚435156653833216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 515913978 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21479834560989 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚515042974313472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 515888306 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚481101386729472 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 214021267 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125211858 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456636471354845 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 203272774 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 214021265 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21915405512704 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚11226.178082128 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 301892711 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚78061592 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚481101632935936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚459559025989632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 301892711 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21915680625664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 203273452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456636712043353 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561¸pæÖm4ø+/ _wandbruntime‚11227  _runtime‚11227Ù›÷kÜ ù+óM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21544426651136 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 125222215 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚515086007785472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21481890609945 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚435192939251712 _runtime‚11227.03345812 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚493168463384064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 515957274 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 515931576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 214038986 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21544426651136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 515957274 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459597381936640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21544198434304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 301918288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 301892592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚78067211 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21481666960589 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 214038984 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 203290055 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 214038984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 301918288 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203289424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456674606212301 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78067840 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚481141808587776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459597363933696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚515085708745728 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚459597381936640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459597363933696 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 301892592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21917269301760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0¥÷fpþ&ú+øM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚435192939251712 _runtime‚11227.03345812 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚493168463384064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 515957274 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 515931576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 214038986 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21544426651136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 515957274 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459597381936640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21544198434304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 301918288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 301892592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚78067211 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21481666960589 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 214038984 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 203290055 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 214038984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 301918288 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203289424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456674606212301 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78067840 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚481141808587776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459597363933696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚515085708745728 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚459597381936640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459597363933696 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 301892592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21917269301760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21544426651136 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 125222215 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚515086007785472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21481890609945¹„ôäè4û+/ _wandbruntime‚11227  _runtime‚11227Ú$¯÷\ü+Ö 2loss/SRqqt4/assignment_loss‚1.1709274297883345 1loss/FBt1/assignment_loss‚0.44533187552709325 1loss/SRqqt2/detection_loss‚0.8939901235315836 /loss/FBt3/detection_loss‚2.5703353732824326 0loss/FRt2/assignment_loss‚1.1794787064194678 /loss/FRt3/assignment_loss‚2.980489822833434 /loss/FRt4/assignment_loss‚3.239960195020188 /loss/FRt2/detection_loss‚1.7490528670139611 0loss/SRqqt1/detection_loss‚2.055043007198133 $loss/total_loss‚39.6045902528373 0loss/FRt1/assignment_loss‚3.2283131769725255 2loss/SRqqt1/assignment_loss‚1.6156446377893812 1loss/SRqqt3/detection_loss‚1.7092100952533966 % _timestamp‚1.7884277562847605e+09 _runtime‚11227.034590954 2loss/SRqqt3/assignment_loss‚1.3085942685287049 1loss/FBt2/assignment_loss‚0.21916703893965883 0loss/FBt3/assignment_loss‚0.2803468683268875 /loss/FRt3/detection_loss‚2.2679311101851254 /loss/FBt4/detection_loss‚1.8034808933734894 -loss/FBt1/detection_loss‚2.48456472158432 trainer/global_step‚11599 2loss/SRqqt2/assignment_loss‚0.8689622632933386 /loss/FRt4/detection_loss‚2.2632801292517883 0loss/SRqqt4/detection_loss‚2.076030885757402 /loss/FBt2/detection_loss‚0.4693364024662027 epoch‚9 0loss/FBt4/assignment_loss‚0.2904096008860506 /loss/FRt1/detection_loss‚2.4347087596143995 _step‚954µ¡Y9aý+Û 2loss/SRqqt3/assignment_loss‚1.3085942685287049 1loss/FBt2/assignment_loss‚0.21916703893965883 0loss/FBt3/assignment_loss‚0.2803468683268875 /loss/FRt3/detection_loss‚2.2679311101851254 /loss/FBt4/detection_loss‚1.8034808933734894 -loss/FBt1/detection_loss‚2.48456472158432 trainer/global_step‚11599 2loss/SRqqt2/assignment_loss‚0.8689622632933386 /loss/FRt4/detection_loss‚2.2632801292517883 0loss/SRqqt4/detection_loss‚2.076030885757402 /loss/FBt2/detection_loss‚0.4693364024662027 epoch‚9 0loss/FBt4/assignment_loss‚0.2904096008860506 /loss/FRt1/detection_loss‚2.4347087596143995 _step‚954 2loss/SRqqt4/assignment_loss‚1.1709274297883345 1loss/FBt1/assignment_loss‚0.44533187552709325 1loss/SRqqt2/detection_loss‚0.8939901235315836 /loss/FBt3/detection_loss‚2.5703353732824326 0loss/FRt2/assignment_loss‚1.1794787064194678 /loss/FRt3/assignment_loss‚2.980489822833434 /loss/FRt4/assignment_loss‚3.239960195020188 /loss/FRt2/detection_loss‚1.7490528670139611 0loss/SRqqt1/detection_loss‚2.055043007198133 $loss/total_loss‚39.6045902528373 0loss/FRt1/assignment_loss‚3.2283131769725255 2loss/SRqqt1/assignment_loss‚1.6156446377893812 1loss/SRqqt3/detection_loss‚1.7092100952533966 % _timestamp‚1.7884277562847605e+09 _runtime‚11227.034590954ºœÍ»Qþ+‚(:… ôûäÔ”©Ùx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.322' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝB£~ÿ+‚(:ó ôûäÔÙǤ{# disk.dm-0.out‚52336.71240234375 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.3671875 disk.sda3.out‚ 513.13671875 disk.rbd1.out‚ 5.58251953125 cpu‚25.291239913802098 network.sent‚ 2.9466378e+07) disk./.usagePercent‚73.73794609005633 disk.dm-1.out‚ 513.13671875 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.rbd3.out‚0$ memory_percent‚97.43163585662842 proc.cpu.threads‚17 disk.sda2.out‚ 0.0390625( disk.nvme0c0n1.out‚52336.71240234375 disk.sda1.out‚0 disk.nvme0c0n1.in‚5.3125! proc.memory.rssMB‚ 2775.234375 network.recv‚ 3.992893e+06 disk.sda2.in‚0 disk.rbd1.in‚0% disk./.usageGB‚1098.7486839294434& disk.nvme0n1.out‚52336.71240234375 disk.dm-1.in‚ 7.31640625 disk.rbd2.in‚0! disk.rbd2.out‚361.85595703125 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 513.17578125 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.rbd4.in‚0 disk.sda3.in‚ 7.31640625( proc.memory.availableMB‚ 420.80078125) proc.memory.percent‚16.93868637084961îÖ{/š€,‚(: ƒüäÔÊæÆx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚97.692' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.538399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö3ĉy,‚(:î ƒüäÔ‘ÁÆz disk.rbd3.out‚0 disk.dm-1.out‚ 513.66015625 disk.sda.in‚ 7.3671875 disk.dm-0.in‚5.3125 network.recv‚ 3.999419e+06) disk./.usagePercent‚73.73794609005633 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.in‚ 0.05078125 disk.sda3.in‚ 7.31640625" disk.dm-0.out‚52435.8134765625 network.sent‚ 2.9575118e+07! disk.rbd2.out‚361.94580078125 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 7.31640625$ memory_percent‚97.43313789367676 cpu‚25.29092237996198 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda.out‚ 513.69921875 disk.sda3.out‚ 513.66015625 disk.rbd1.out‚ 5.58251953125) proc.memory.percent‚16.93868637084961' disk.nvme0c0n1.out‚52435.8134765625% disk.nvme0n1.out‚52435.8134765625' proc.memory.availableMB‚ 420.5546875! proc.memory.rssMB‚ 2775.234375 disk.nvme0n1.in‚5.3125 disk.sda2.out‚ 0.0390625% disk./.usageGB‚1098.7486839294434 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd4.out‚0(Ui‚,‚(:… ’üäÔöµ¼x gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.322' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`¦uÖyƒ,‚(:î ’üäÔ÷ŸŽ{ disk.nvme0n1.in‚5.3125 disk.rbd0.out‚0 disk.rbd4.in‚0! proc.memory.rssMB‚ 2775.234375 cpu‚25.290577565075726 disk.rbd1.out‚ 5.58251953125 disk.rbd2.out‚363.6142578125 disk.sda.in‚ 7.3671875' proc.memory.availableMB‚ 420.5703125 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda3.in‚ 7.31640625) disk./.usagePercent‚73.73794609005633" disk.dm-0.out‚52436.1787109375 disk.sda2.in‚0 disk.drbd1777.in‚ 1.03515625' disk.nvme0c0n1.out‚52436.1787109375 disk.sda1.in‚ 0.05078125 disk.rbd3.in‚0 network.sent‚ 2.9580956e+07% disk.nvme0n1.out‚52436.1787109375 disk.sda2.out‚ 0.0390625 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 514.06640625$ memory_percent‚97.43304252624512 disk.rbd1.in‚0 disk.dm-0.in‚5.3125 disk.rbd0.in‚0 disk.sda.out‚ 514.10546875 proc.cpu.threads‚17 disk.dm-1.out‚ 514.06640625 disk.rbd2.in‚0) proc.memory.percent‚16.93868637084961 network.recv‚ 4.003696e+06% disk./.usageGB‚1098.7486839294434 disk.dm-1.in‚ 7.31640625 disk.rbd3.out‚0¡@”~4„,/ _wandbruntime‚11271  _runtime‚11271ØË…,Å ' lr-AdamW/pg1‚3.2474269951248724e-06 ' lr-AdamW/pg2‚3.2474269951248724e-06 trainer/global_step‚11649 % _timestamp‚1.7884278008322318e+09 _runtime‚11271.582164342 _step‚955ðPòíІ,Ê _step‚955 ' lr-AdamW/pg1‚3.2474269951248724e-06 ' lr-AdamW/pg2‚3.2474269951248724e-06 trainer/global_step‚11649 % _timestamp‚1.7884278008322318e+09 _runtime‚11271.582164342»UR¦û4‡,/ _wandbruntime‚11271  _runtime‚11271J5±bWˆ,êO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 518055158 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 303173613 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21634488080384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚495185778500608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 214907215 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚483110429276160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 303173613 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204112620 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 303147943 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚483110183069696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚458543227818853 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 214907217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307441664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21634259876864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 214907217 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21634488080384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21571691781345 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚436971776726016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚78383010 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125729608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚78382332 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚458543468507361 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚436971759686656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517194464711680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚483110429276160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 283501568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚461475923192832 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 214907215 dQDeviceStatsMonitor.on_train_batch_st NöÇ™art/inactive_split_bytes.small_pool.allocated‚22008993652736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.788427800832912e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚461475923192832 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22008710151168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 _runtime‚11271.583032858 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚483110183069696 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚483110183069696 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚483110183069696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚458543227818853 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 214907217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307441664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21634259876864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 214907217 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21634488080384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21571691781345 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚436971776726016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚78383010 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125729608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚78382332 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚458543468507361 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚436971759686656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517194464711680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚483110429276160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 283501568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚461475923192832 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 214907215 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22008993652736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0¼Œ3Ôª4Š,/ _wandbruntime‚11272  _runtime‚11272–LOñù&‹,óM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚461514261136896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21636325788160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78389274 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚437008062144512 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 518124126 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21573524181153 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 518124126 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 214924936 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 204129239 _step‚957 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204128625 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21636325788160 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 214924934 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 125739963 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚495226948196864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚461514279139840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 214924934 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 518098428 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 303199190 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚458581603365025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚437008079183872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 518098428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚495226924256768 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚461514261136896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22010585510912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚483150604928000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 303199190 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚483150604928000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21636097571328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚458581362676309 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0]#*…þ&Œ,øM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚461514279139840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 214924934 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 290651136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 518098428 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 303199190 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚458581603365025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚437008079183872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 518098428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚495226924256768 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚461514261136896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22010585510912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚483150604928000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 303199190 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚483150604928000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21636097571328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚458581362676309 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚461514261136896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21636325788160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78389274 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚437008062144512 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 518124126 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21573524181153 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 518124126 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 214924936 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 204129239 _step‚957 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204128625 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21636325788160 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 214924934 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 125739963 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚495226948196864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824½¶ãÿ±4,/ _wandbruntime‚11272  _runtime‚11272}ÍöâŽ,× 1loss/SRqqt4/detection_loss‚2.0414924883842467 $ _timestamp‚1.788427801701203e+09 1loss/FBt4/assignment_loss‚0.35916976902323466 /loss/FBt1/detection_loss‚2.6749986886978148 %loss/total_loss‚39.820545# Îü{32346565 _runtime‚11272.451033889 0loss/FRt3/assignment_loss‚2.4658973464151708 2loss/SRqqt2/assignment_loss‚0.9635272274408907 1loss/FBt1/assignment_loss‚0.07236315597110661 .loss/FBt2/detection_loss‚0.536625109771948 epoch‚9 0loss/FRt4/assignment_loss‚3.7815646141767503 2loss/SRqqt1/assignment_loss‚1.3199017605345165 0loss/FBt2/assignment_loss‚0.1474746878110817 _step‚958 /loss/FRt2/detection_loss‚1.3561614375927897 .loss/FRt3/detection_loss‚2.397596560600327 /loss/FRt4/detection_loss‚2.5373588964398506 0loss/FRt1/assignment_loss‚2.0664688414997525 .loss/FBt3/detection_loss‚2.379712621370951 trainer/global_step‚11649 2loss/SRqqt3/assignment_loss‚1.1886518378835171 0loss/FBt3/assignment_loss‚0.4800391487175754 /loss/FBt4/detection_loss‚2.8569571574529014 0loss/FRt2/assignment_loss‚1.0140173708806273 2loss/SRqqt4/assignment_loss‚1.5271773010492324 0loss/SRqqt1/detection_loss‚2.204928138426372 1loss/SRqqt3/detection_loss‚1.6921423161029816 .loss/FRt1/detection_loss‚2.875755264803215 1loss/SRqqt2/detection_loss‚0.8805635824187923â)›b,Ü trainer/global_step‚11649 2loss/SRqqt3/assignment_loss‚1.1886518378835171 0loss/FBt3/assignment_loss‚0.4800391487175754 /loss/FBt4/detection_loss‚2.8569571574529014 0loss/FRt2/assignment_loss‚1.0140173708806273 2loss/SRqqt4/assignment_loss‚1.5271773010492324 0loss/SRqqt1/detection_loss‚2.204928138426372 1loss/SRqqt3/detection_loss‚1.6921423161029816 .loss/FRt1/detection_loss‚2.875755264803215 1loss/SRqqt2/detection_loss‚0.8805635824187923 1loss/SRqqt4/detection_loss‚2.0414924883842467 $ _timestamp‚1.788427801701203e+09 1loss/FBt4/assignment_loss‚0.35916976902323466 /loss/FBt1/detection_loss‚2.6749986886978148 %loss/total_loss‚39.82054532346565 _runtime‚11272.451033889 0loss/FRt3/assignment_loss‚2.4658973464151708 2loss/SRqqt2/assignment_loss‚0.9635272274408907 1loss/FBt1/assignment_loss‚0.07236315597110661 .loss/FBt2/detection_loss‚0.536625109771948 epoch‚9 0loss/FRt4/assignment_loss‚3.7815646141767503 2loss/SRqqt1/assignment_loss‚1.3199017605345165 0loss/FBt2/assignment_loss‚0.1474746878110817 _step‚958 /loss/FRt2/detection_loss‚1.3561614375927897 .loss/FRt3/detection_loss‚2.397596560600327 /loss/FRt4/detection_loss‚2.5373588964398506 0loss/FRt1/assignment_loss‚2.0664688414997525 .loss/FBt3/detection_loss‚2.379712621370951¾#¨¯,‚(:„ ¡üäÔž¡x gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.797' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿Äm l‘,‚(:á ¡üäÔ¥›÷z" disk.nvme0n1.out‚ 52536.0859375 disk.rbd2.in‚0 network.sent‚ 2.9587415e+07 disk.sda1.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125) disk./.usagePercent‚73.73794634606385 disk.rbd3.in‚0 cpu‚25.290259240971274 disk.sda.out‚ 514.7421875 disk.rbd1.out‚ 5.58251953125 disk.sda2.in‚0! proc.memory.rssMB‚ 2775.234375% disk./.usageGB‚1098.7486877441406 disk.drbd1777.in‚ 1.03515625$ disk.nvme0c0n1.out‚ 52536.0859375 disk.dm-1.out‚ 514.703125 disk.rbd2.out‚365.2119140625 disk.sda.in‚ 7.3671875 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 514.703125 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0( proc.memory.availableMB‚ 420.55078125 disk.rbd4.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.dm-0.in‚5.3125 disk.dm-0.out‚ 52536.0859375 disk.rbd0.out‚0) proc.memory.percent‚16.93868637084961 disk.nvme0c0n1.in‚5.3125 disk.dm-1.in‚ 7.31640625$ memory_percent‚97.43316173553467 proc.cpu.threads‚17 network.recv‚ 4.008621e+06 disk.drbd1777.out‚0 disk.rbd4.out‚0M£gÊŽ’,‚(:ƒ °üäÔßÀ‰x gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02s”ì}“,‚(:ò °üäԢͫz disk.sda3.out‚ 515.19921875 cpu‚25.289943672467857 disk.rbd4.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0& disk.nvme0n1.out‚52635.49853515625$ memory_percent‚97.43316173553467% disk./.usageGB‚1098.7488098144531 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 disk.dm-1.in‚ 7.31640625( proc.memory.availableMB‚ 420.55078125 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.sda.out‚ 515.23828125 disk.sda3.in‚ 7.31640625 disk.dm-1.out‚ 515.19921875 disk.rbd2.out‚365.2900390625) proc.memory.percent‚16.93868637084961 network.recv‚ 4.013975e+06 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda.in‚ 7.3671875 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.rbd1.out‚ 5.58251953125 disk.dm-0.in‚5.3125# disk.dm-0.out‚52635.49853515625! proc.memory.rssMB‚ 2775.234375) disk./.usagePercent‚73.73795453830495 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 network.sent‚ 2.9695312e+07( disk.nvme0c0n1.out‚52635.49853515625òD/›”,‚(: ¿üäÔŸÄx gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûT–l•,‚(:á ¿üäÔèåÑz disk.rbd3.in‚0 disk.nvme0n1.in‚5.3125! proc.memory.rssMB‚ 2775.234375 cpu‚25.28960274379815 disk.rbd0.out‚0 network.sent‚ 2.9701329e+07 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.rbd4.in‚0% disk.nvme0c0n1.out‚52735.11328125 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0% disk./.usageGB‚1098.7487297058105 disk.rbd2.out‚366.8291015625 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 disk.sda.out‚ 516.140625 disk.sda3.out‚ 516.1015625 disk.rbd1.out‚ 5.58251953125 network.recv‚ 4.018364e+06 disk.rbd4.out‚0# disk.nvme0n1.out‚52735.11328125 disk.dm-0.out‚52735.11328125# memory_percent‚97.4339485168457& proc.memory.availableMB‚ 420.421875) proc.memory.percent‚16.93868637084961 proc.cpu.threads‚17 disk.sda3.in‚ 7.31640625) disk./.usagePercent‚73.73794916214673 disk.dm-1.in‚ 7.31640625 disk.rbd0.in‚0 disk.sda.in‚ 7.3671875 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 516.1015625 disk.nvme0c0n1.in‚5.3125+}(…4–,/ _wandbruntime‚11317  _runtime‚11317}3×2Æ—,À % lr-AdamW/pg2‚2.55353177845675e-06 trainer/global_step‚11699 $ _timestamp‚1.788427846257953e+09 _runtime‚11317.007974311 _step‚959 % lr-AdamW/pg1‚2.55353177845675e-068ñŸ0˘,Å % lr-AdamW/pg1‚2.55353177845675e-06 % lr-AdamW/pg2‚2.55353177845675e-06 trainer/global_step‚11699 $ _timestamp‚1.788427846257953e+09 _runtime‚11317.007974311 _step‚959¿Ì.Ay4™,/ _wandbruntime‚11317  _runtime‚11317Il-ñ'š,ëO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚485119225616384 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 304428845 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884278462586787e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚463392820396032 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126247358 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21663101703405 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚519346322506752 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚460450224971369 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485118979409920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 _step‚960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚463392838398976 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 304428845 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 520247682 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22102356149248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22102083133440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 215793167 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21726387217408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 215793165 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21726159013888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438786882579456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438786899618816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21663325352553 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚661 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚78704069 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚519346619462656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 215793165 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚463392820396032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 215793167 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204952088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚497244239373312 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485118979409920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 204951427 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚78704728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚497244263313408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _runtime‚11317.008801168 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 304454515 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21726159013888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 304454515 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432Å«›ö'›,ðO BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204952088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚497244239373312 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485118979409920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 204951427 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚78704728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚497244263313408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _runtime‚11317.008801168 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 304454515 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21726159013888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 304454515 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚485119225616384 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 304428845 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 273015808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884278462586787e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚463392820396032 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126247358 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21663101703405 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚519346322506752 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚460450224971369 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485118979409920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 _step‚960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚463392838398976 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 304428845 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 520247682 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22102356149248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22102083133440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 215793167 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21726387217408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 215793165 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21726159013888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438786882579456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438786899618816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21663325352553 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚661 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚78704069 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚519346619462656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 215793165 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚463392820396032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 215793167À‘f-4œ,/ _wandbruntime‚11317  _runtime‚11317kŠS’,õM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚463431176343040 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 204969144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 304454396 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21727996708352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78711429 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 trainer/global_step‚11699 % _timestamp‚1.7884278471155434e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚519389354435584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21665157752361 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21664934103005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21728224925184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 304480092 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚463431158340096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126257713 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 520265280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 520290978 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚519389653475328 _runtime‚11317.866168303 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21727996708352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 SDDeviceStat5æHisMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚460488119140317 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 215810884 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204968484 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 304480092 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21728224925184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 520290978 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚460488359829033 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _step‚961 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438823185037312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22103945366016 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126257715 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485159155048448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438823202076672dìÿç'ž,úM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚519389653475328 _runtime‚11317.866168303 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21727996708352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚460488119140317 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 215810884 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204968484 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 304480092 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21728224925184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 520290978 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚460488359829033 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 _step‚961 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275099648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438823185037312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22103945366016 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126257715 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485159155048448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438823202076672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚463431176343040 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 204969144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 304454396 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21727996708352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78711429 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299039744 trainer/global_step‚11699 % _timestamp‚1.7884278471155434e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚519389354435584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21665157752361 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21664934103005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21728224925184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 304480092 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚463431158340096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126257713 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 520265280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 520290978 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299Áe T¨4Ÿ,/ _wandbruntime‚11317  _runtime‚11317µ-ra ,Û 1loss/SRqqt2/detection_loss‚0.8973118972735129 /loss/FBt2/detection_loss‚0.5093628062620863 1loss/FBt2/assignment_loss‚0.06586736689437107 /loss/FRt3/detection_loss‚2.4920280252365354 /loss/FBt1/detection_loss‚3.1937793612480165 epoch‚9 _runtime‚11317.867427075 0loss/FRt2/assignment_loss‚1.1877061907125979 /loss/FRt2/detection_loss‚1.6344211337213614 1loss/SRqqt3/detection_loss‚2.1233619345085963 0loss/FBt3/assignment_loss‚0.5653851848328486 %loss/total_loss‚42.11225605306565 2loss/SRqqt1/assignment_loss‚1.6229117212862503 2loss/SRqqt4/assignment_loss‚1.6734731515674894 /loss/FRt4/detection_loss‚1.9680284126463925 % _timestamp‚1.7884278471175756e+09 _step‚962 0loss/FRt3/assignment_loss‚2.9269636298747774 1loss/SRqqt4/detection_loss‚1.7405439172662907 2loss/SRqqt3/assignment_loss‚1.4117948707370531 1loss/FBt1/assignment_loss‚0.04694258480631106 0loss/FRt4/assignment_loss‚2.9480420675194052 2loss/SRqqt2/assignment_loss‚0.6859848040790446 /loss/FRt1/detection_loss‚2.5678611288175865 1loss/SRqqt1/detection_loss‚2.3285998047851937 .loss/FBt3/detection_loss‚3.402704209089279 /loss/FBt4/detection_loss‚2.1772160728772483 trainer/global_step‚11699 /loss/FRt1/assignment_loss‚3.298498015193378 0loss/FBt4/assignment_loss‚0.6434677618300257‹ŸpHf¡,à /loss/FRt1/detection_loss‚2.5678611288175865 1loss/SRqqt1/detection_loss‚2.3285998047851937 .loss/FBt3/detection_loss‚3.402704209089279 /loss/FBt4/detection_loss‚2.1772160728772483 trainer/global_step‚11699 /loss/FRt1/assignment_loss‚3.298498015193378 0loss/FBt4/assignment_loss‚0.6434677618300257 1loss/SRqqt2/detection_loss‚0.8973118972735129 /loss/FBt2/detection_loss‚0.5093628062620863 1loss/FBt2/assignment_loss‚0.06586736689437107 /loss/FRt3/detection_loss‚2.4920280252365354 /loss/FBt1/detection_loss‚3.1937793612480165 epoch‚9 _runtime‚11317.867427075 0loss/FRt2/assignment_loss‚1.1877061907125979 /loss/FRt2/detection_loss‚1.6344211337213614 1loss/SRqqt3/detection_loss‚2.1233619345085963 0loss/FBt3/assignment_loss‚0.5653851848328486 %loss/total_loss‚42.11225605306565 2loss/SRqqt1/assignment_loss‚1.6229117212862503 2loss/SRqqt4/assignment_loss‚1.6734731515674894 /loss/FRt4/detection_loss‚1.9680284126463925 % _timestamp‚1.7884278471175756e+09 _step‚962 0loss/FRt3/assignment_loss‚2.9269636298747774 1loss/SRqqt4/detection_loss‚1.7405439172662907 2loss/SRqqt3/assignment_loss‚1.4117948707370531 1loss/FBt1/assignment_loss‚0.04694258480631106 0loss/FRt4/assignment_loss‚2.9480420675194052 2loss/SRqqt2/assignment_loss‚0.6859848040790446ÂñˆÍ¢,‚(:„ ÎüäÔ˜±«x gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.541' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5082 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùìªv£,‚(:ë ÎüäÔ¼úüz disk.rbd0.in‚0( proc.memory.availableMB‚ 420.63671875) proc.memory.percent‚16.93868637084961 network.sent‚ 2.97072e+07 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚ 7.31640625$ memory_percent‚97.43263721466064 network.recv‚ 4.022438e+06 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.sda3.out‚ 516.6171875! proc.memory.rssMB‚ 2775.234375 disk.nvme0n1.in‚5.3125 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.3671875% disk./.usageGB‚1098.7487182617188 disk.drbd1777.out‚0) disk./.usagePercent‚73.73794839412413 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 7.31640625& disk.nvme0n1.out‚52834.27587890625 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 516.65625# disk.dm-0.out‚52834.27587890625( disk.nvme0c0n1.out‚52834.27587890625 cpu‚25.28928455006756 proc.cpu.threads‚17 disk.rbd1.out‚ 5.58251953125 disk.dm-1.out‚ 516.6171875! disk.rbd2.out‚366.98193359375xÀ£¤,‚(:„ ÝüäÔ©ÕÞx gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.681' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôHˆJw¥,‚(:ì ÝüäÔÌÌèz disk.sda.in‚ 7.3671875 disk.sda3.in‚ 7.31640625( proc.memory.availableMB‚ 420.39453125! disk.dm-0.out‚52834.775390625 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 network.sent‚ 2.9816382e+07 disk.rbd1.out‚ 5.66845703125$ disk.nvme0n1.out‚52834.775390625 disk.sda2.in‚0 disk.sda3.out‚ 517.17578125 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125! disk.rbd2.out‚368.52099609375 disk.rbd3.in‚0% disk./.usageGB‚1098.7487182617188 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 disk.sda1.in‚ 0.05078125 network.recv‚ 4.029234e+06) disk./.usagePercent‚73.73794839412413 disk.rbd1.in‚0 disk.dm-1.out‚ 517.17578125 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚52834.775390625 disk.dm-1.in‚ 7.31640625! proc.memory.rssMB‚ 2775.234375 disk.sda2.out‚ 0.0390625 disk.sda.out‚ 517.21484375$ memory_percent‚97.43411540985107) proc.memory.percent‚16.93868637084961 cpu‚25.28896952682413 disk.drbd1777.in‚ 1.03515625»¦”¶¦,‚(:… ìüäÔ·ÿŽx gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{»FÈj§,‚(:ß ìüäÔÛî¸{ disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 network.sent‚ 2.9820747e+07 disk.sda3.in‚ 7.31640625 disk.rbd1.in‚0 disk.nvme0n1.in‚5.3125 disk.dm-1.in‚ 7.31640625 disk.sda1.in‚ 0.05078125 cpu‚25.28862972600252) disk./.usagePercent‚73.73795325826728% disk./.usageGB‚1098.7487907409668 disk.drbd1777.out‚0 disk.sda.in‚ 7.3671875 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.dm-1.out‚517.75! disk.rbd2.out‚370.17333984375 disk.rbd4.out‚0! proc.memory.rssMB‚ 2775.234375 network.recv‚ 4.032098e+06 disk.rbd3.in‚0! disk.dm-0.out‚52933.873046875 disk.rbd2.in‚0 disk.drbd1777.in‚ 1.03515625$ disk.nvme0n1.out‚52933.873046875 disk.rbd0.in‚0$ memory_percent‚97.43263721466064 proc.cpu.threads‚17 disk.sda.out‚ 517.7890625 disk.rbd1.out‚ 5.73095703125 disk.sda2.out‚ 0.0390625 disk.rbd0.out‚0& disk.nvme0c0n1.out‚52933.873046875( proc.memory.availableMB‚ 420.63671875) proc.memory.percent‚16.93868637084961 disk.rbd3.out‚0 disk.sda3.out‚517.75 disk.rbd4.in‚0²Z4¨,/ _wandbruntime‚11362  _runtime‚113620•øË©,Å % _timestamp‚1.7884278916483557e+09 _runtime‚11362.398337469 _step‚963 ' lr-AdamW/pg1‚1.9427054138455884e-06 ' lr-AdamW/pg2‚1.9427054138455884e-06 trainer/global_step‚11749à¬-Ъ,Ê % _timestamp‚1.7884278916483557e+09 _runtime‚11362.398337469 _step‚963 ' lr-AdamW/pg1‚1.9427054138455884e-06 ' lr-AdamW/pg2‚1.9427054138455884e-06 trainer/global_step‚11749Ë€ß4«,/ _wandbruntime‚11362  _runtime‚11362[s; $$¬,ëO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 522388862 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 _runtime‚11362.399317045 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21818286354432 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚465309735602176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚11749 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21818058150912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 305709747 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚499302748126208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚465309717599232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79026287 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚440602022511616 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚487128021956608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚487128021956608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚465309717599232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚79026948 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21754735274613 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 305709747 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 205792058 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21818058150912 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 205791395 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚440602005472256 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _step‚964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 216679117 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚521498283968000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 305735417 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚487127775750144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 216679115 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 305735417 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21754958923761 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 522414534 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 216679117 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_byÎpîÍtes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126765110 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚499302724186112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21818286354432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚465309735602176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608d «¬ö'­,ðO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚440602022511616 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚487128021956608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚487128021956608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚465309717599232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚79026948 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21754735274613 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 305709747 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 205792058 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21818058150912 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 205791395 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚440602005472256 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 _step‚964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 216679117 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚521498283968000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 305735417 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚487127775750144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 216679115 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 305735417 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21754958923761 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 522414534 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 216679117 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126765110 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚499302724186112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21818286354432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚465309735602176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 522388862 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 _runtime‚11362.399317045 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21818286354432 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚465309735602176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚11749 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21818058150912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 305709747 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚499302748126208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚465309717599232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79026287 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299ÄÞ”zc4®,/ _wandbruntime‚11363  _runtime‚11363û·Óú&¯,ôM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 305735298 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126775465 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚465348055543296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 522432132 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788427892500435e+09 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205807932 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 305735298 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚487168197608448 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚465348055543296 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79032469 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 258322432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 216696836 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚465348073546240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 216696834 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚521541331242496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚499343917822464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 216696834 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126775463 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚440638324969472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚487168197608448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 522457830 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11749 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 205808590 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79033125 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ƒ4'ÿ&°,ùM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 _step‚965 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205807932 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 305735298 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚487168197608448 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚465348055543296 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79032469 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 258322432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 216696836 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚465348073546240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 216696834 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚521541331242496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚499343917822464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 216696834 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126775463 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚440638324969472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚487168197608448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 522457830 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11749 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 205808590 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79033125 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 305735298 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126775465 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚465348055543296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 522432132 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788427892500435e+09 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 524555714 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 217565067 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442417128365056 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚489136818296832 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21846592494969 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523650224432128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467226614802432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚663 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚501361232939008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22289015433216 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21910185491456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 206634930 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚489136572090368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚467226614802432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 217565065 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489136572090368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467226632805376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚79352735 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 306990649 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 206635595 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚665 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚489136818296832·M´Uö'¿,ðO dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22289280060416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 264627200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚464263737899385 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 524555714 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 217565067 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442417128365056 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚489136818296832 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21846592494969 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523650224432128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467226614802432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚663 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚501361232939008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22289015433216 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21910185491456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 206634930 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚489136572090368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚467226614802432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 217565065 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489136572090368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467226632805376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚79352735 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 306990649 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 206635595 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚665 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚489136818296832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21909957287936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464263497210877 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 307016319 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21910185491456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127282860 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 trainer/global_step‚11799 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21909957287936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 217565065 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 _step‚968È€v·4À,/ _wandbruntime‚11408  _runtime‚11408*é8WÁ,õM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 524624682 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 524598984 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127293215 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚501402402635264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚442453430822912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 524598984 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚523693561181696 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206651451 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464301632068333 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 307041896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22290885543936 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79358871 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22291158546432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚467264970749440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127293213 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚489176993948672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 307016200 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚467264952746496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚11408.550797403 _step‚969 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21911794982400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 217582784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 524624682 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21912023199232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 217582786 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21911794982400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 307016200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11799 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 206652086 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 217582786 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚489176747728896 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚501402378695168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21848424894777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚442453447862272 % _timestamp‚1.7884279378003833e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21848201245421utŠ/'Â,úM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79358871 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22291158546432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚467264970749440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127293213 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚489176993948672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 307016200 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚467264952746496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚11408.550797403 _step‚969 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21911794982400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 217582784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 296942592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 524624682 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21912023199232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 217582786 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21911794982400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 307016200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11799 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 206652086 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 217582786 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚489176747728896 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚501402378695168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21848424894777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚442453447862272 % _timestamp‚1.7884279378003833e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21848201245421 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 524624682 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 524598984 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127293215 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚501402402635264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚442453430822912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 524598984 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚523693561181696 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206651451 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464301632068333 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 307041896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22290885543936ÉtD24Ã,/ _wandbruntime‚11408  _runtime‚11408ùbWÝaÄ,Û 0loss/FRt3/assignment_loss‚2.4956483182154203 1loss/FBt2/assignment_loss‚0.08854227102043428 0loss/FBt3/assignment_loss‚0.1610463851131499 /loss/FRt1/detection_loss‚3.5026881491596047 /loss/FRt2/detection_loss‚1.6571359796256855 1loss/SRqqt3/detection_loss‚1.7791580610415514 epoch‚9 2loss/SRqqt3/assignment_loss‚1.2352181992986624 /loss/FRt3/detection_loss‚2.7485652542427967 .loss/FBt4/detection_loss‚2.996935319900513 &loss/total_loss‚42.007604472728616 % _timestamp‚1.7884279378020768e+09 2loss/SRqqt2/assignment_loss‚0.7564172706395191 /loss/FBt4/assignment_loss‚0.222328294464387 1loss/SRqqt1/detection_loss‚2.7443117653501443 1loss/SRqqt2/detection_loss‚0.9009327299794868 /loss/FBt1/detection_loss‚1.8446932766172621 /loss/FBt3/detection_loss‚2.8000468969345094 0loss/FRt1/assignment_loss‚3.7871257825331255 1loss/FBt1/assignment_loss‚0.15726025921442974 .loss/FRt4/detection_loss‚2.318625614481027 1loss/SRqqt4/detection_loss‚1.7527507828905227 /loss/FBt2/detection_loss‚0.4391152057279315 0loss/FRt2/assignment_loss‚1.3545804342319225 2loss/SRqqt1/assignment_loss‚1.9340399138886353 2loss/SRqqt4/assignment_loss‚0.9707577414810658 _runtime‚11408.551896674 0loss/FRt4/assignment_loss‚3.3596805666768272 trainer/global_step‚11799 _step‚970ŒìéIfÅ,à 1loss/FBt1/assignment_loss‚0.15726025921442974 .loss/FRt4/detection_loss‚2.318625614481027 1loss/SRqqt4/detection_loss‚1.7527507828905227 /loss/FBt2/detection_loss‚0.4391152057279315 0loss/FRt2/assignment_loss‚1.3545804342319225 2loss/SRqqt1/assignment_loss‚1.9340399138886353 2loss/SRqqt4/assignment_loss‚0.9707577414810658 _runtime‚11408.551896674 0loss/FRt4/assignment_loss‚3.3596805666768272 trainer/global_step‚11799 _step‚970 0loss/FRt3/assignment_loss‚2.4956483182154203 1loss/FBt2/assignment_loss‚0.08854227102043428 0loss/FBt3/assignment_loss‚0.1610463851131499 /loss/FRt1/detection_loss‚3.5026881491596047 /loss/FRt2/detection_loss‚1.6571359796256855 1loss/SRqqt3/detection_loss‚1.7791580610415514 epoch‚9 2loss/SRqqt3/assignment_loss‚1.2352181992986624 /loss/FRt3/detection_loss‚2.7485652542427967 .loss/FBt4/detection_loss‚2.996935319900513 &loss/total_loss‚42.007604472728616 % _timestamp‚1.7884279378020768e+09 2loss/SRqqt2/assignment_loss‚0.7564172706395191 /loss/FBt4/assignment_loss‚0.222328294464387 1loss/SRqqt1/detection_loss‚2.7443117653501443 1loss/SRqqt2/detection_loss‚0.9009327299794868 /loss/FBt1/detection_loss‚1.8446932766172621 /loss/FBt3/detection_loss‚2.8000468969345094 0loss/FRt1/assignment_loss‚3.7871257825331255Ê"sÆ,‚(:… ¨ýäÔ¶å«x gpu.0.gpu‚21.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ\ƒÜmÇ,‚(:â ¨ýäÔÔïäz disk.sda3.in‚ 7.31640625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.in‚ 7.3671875 disk.dm-1.in‚ 7.31640625 disk.dm-1.out‚ 520.1953125) proc.memory.percent‚16.93868637084961 cpu‚25.287381338776% disk./.usageGB‚1098.7493591308594 disk.sda3.out‚ 520.1953125 disk.rbd2.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2775.234375 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.drbd1777.out‚0 disk.rbd4.out‚0# memory_percent‚97.4332332611084 disk.rbd1.out‚ 5.826171875$ disk.nvme0n1.out‚53232.884765625 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd3.in‚0! disk.dm-0.out‚53232.884765625 disk.sda2.out‚ 0.0390625! disk.rbd2.out‚373.32568359375) disk./.usagePercent‚73.73799140338988 disk.sda1.out‚0 disk.sda.out‚ 520.234375 disk.rbd0.in‚0' proc.memory.availableMB‚ 420.5390625 disk.rbd1.in‚0 disk.sda1.in‚ 0.05078125 disk.rbd3.out‚0 network.sent‚ 2.9947035e+07 network.recv‚ 4.050825e+06 disk.drbd1777.in‚ 1.03515625& disk.nvme0c0n1.out‚53232.884765625« ‚ŽÈ,‚(:ƒ ·ýäÔÊÖ³x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.68' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÊ¡|É,‚(:ñ ·ýäÔà”Òz network.sent‚ 3.0055725e+07# disk.dm-0.out‚53331.99462890625 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚ 7.31640625 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚0! proc.memory.rssMB‚ 2775.234375% disk./.usageGB‚1098.7493591308594 disk.rbd1.out‚ 5.826171875 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.3671875 disk.sda3.out‚ 520.71484375 disk.rbd1.in‚0 disk.dm-0.in‚5.3125! disk.rbd2.out‚373.48974609375$ memory_percent‚97.43316173553467) disk./.usagePercent‚73.73799140338988 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda.out‚ 520.75390625 disk.sda3.in‚ 7.31640625( proc.memory.availableMB‚ 420.55078125) proc.memory.percent‚16.93868637084961 cpu‚25.287049375566493 disk.sda1.in‚ 0.05078125 network.recv‚ 4.057439e+06 disk.sda2.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚53331.99462890625 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚53331.99462890625 disk.dm-1.out‚ 520.71484375 disk.rbd2.in‚0ìôCAÊ,‚(:„ ÆýäÔÂܵy gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…U\DpË,‚(:å ÆýäÔöÒƒ{ disk.sda.out‚ 521.2265625 disk.rbd1.out‚ 5.88916015625) proc.memory.percent‚16.93868637084961 network.sent‚ 3.0061612e+07' disk.nvme0c0n1.out‚53431.2744140625 disk.sda.in‚ 7.3671875" disk.dm-0.out‚53431.2744140625 disk.dm-1.in‚ 7.31640625! proc.memory.rssMB‚ 2775.234375 cpu‚25.286737633147816) disk./.usagePercent‚73.73798961133714 disk.rbd3.in‚0 disk.sda2.in‚0 network.recv‚ 4.061601e+06 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚ 7.31640625% disk.nvme0n1.out‚53431.2744140625% disk./.usageGB‚1098.7493324279785 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚5.3125 disk.sda1.out‚0 disk.rbd1.in‚0# memory_percent‚97.4332332611084! disk.rbd2.out‚375.00146484375 disk.drbd1777.out‚0 disk.nvme0n1.in‚5.3125 disk.dm-0.in‚5.3125 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda3.out‚521.1875 disk.sda2.out‚ 0.0390625' proc.memory.availableMB‚ 420.5390625 proc.cpu.threads‚17 disk.dm-1.out‚521.1875 disk.rbd4.out‚0 í14Ì,/ _wandbruntime‚11453  _runtime‚11453Œ¿ÈÉÍ,à & lr-AdamW/pg1‚9.707510840372215e-07 & lr-AdamW/pg2‚9.707510840372215e-07 trainer/global_step‚11849 % _timestamp‚1.7884279822700157e+09 _runtime‚11453.019979632 _step‚971ðÙ^ ÎÎ,È & lr-AdamW/pg1‚9.707510840372215e-07 & lr-AdamW/pg2‚9.707510840372215e-07 trainer/global_step‚11849 % _timestamp‚1.7884279822700157e+09 _runtime‚11453.019979632 _step‚971Ëÿß´4Ï,/ _wandbruntime‚11453  _runtime‚11453àŠÏñ'Ð,ëO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚444232251257856 _step‚972 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 308271551 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚525802424052224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21938002417029 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491145614637056 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22382426993152 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 526722566 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 526748238 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 % _timestamp‚1.7884279822708418e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚469143512005632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 218451015 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚503419693811712 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚469143530008576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491145368430592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 218451015 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79672867 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚466170253674885 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 308271551 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22001856424960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22001856424960 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 207474133 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚466170494363393 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 308297221 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127800608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127800610 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚223827063004165EBôÑ,ðO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 QBDev ,`E'iceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22002084628480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚525802120804864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚503419717751808 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11849 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚469143530008576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491145368430592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 218451015 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79672867 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚466170253674885 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 308271551 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22001856424960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22001856424960 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 207474133 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚466170494363393 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 308297221 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127800608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127800610 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22382706300416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚444232251257856 _step‚972 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 308271551 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚525802424052224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21938002417029 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491145614637056 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22382426993152 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 526722566 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 526748238 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 % _timestamp‚1.7884279822708418e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚469143512005632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 218451015 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚503419693811712 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚469181867952640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 218468736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚444268553715712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 _runtime‚11453.874425292 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 308322798 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 526765836 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 308297102 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 207490019 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127810965 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚525845157656064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚503460863507968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚469181867952640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21940058465985 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 218468734 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚503460887448064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 526765836 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚469181849949696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚469181849949696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 526791534 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 218468736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884279831237674e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79679056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚525845452501504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 308297102 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79679699 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 _step‚973 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 218468734 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚503460887448064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 526765836 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚469181849949696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚469181849949696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 526791534 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 218468736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884279831237674e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79679056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚525845452501504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 308297102 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79679699 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 _step‚973 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚469181867952640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 218468736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚444268553715712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 _runtime‚11453.874425292 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 308322798 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 526765836 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 308297102 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 207490019 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127810965 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚525845157656064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚503460863507968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚469181867952640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21940058465985 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚493154164770816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 208310588 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚446047391190016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚471060409208832 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22093983765504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 219336967 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚471060427211776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79992230 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚493154164770816 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 309552453 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 128318360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 _runtime‚11498.115287626 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 528889418 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 528915090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _step‚976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 219336965 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 309552453 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚527954269207552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 $ _timestamp‚1.788428027365139e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚505478202564608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 309578123 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 208311229 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚505478178624512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 219336965 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚468077250827401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚493154410977280 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚493154164770816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 208310588 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚446047391190016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚471060409208832 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22093983765504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 219336967 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚471060427211776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79992230 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚493154164770816 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 309552453 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 128318360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 _runtime‚11498.115287626 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 528889418 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 528915090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292761600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 _step‚976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 219336965 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 309552453 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚527954269207552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 $ _timestamp‚1.788428027365139e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚505478202564608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 309578123 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 208311229 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚505478178624512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 219336965 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚468077250827401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚493154410977280 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 128328713 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22031468387837 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79998681 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 309603700 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 219354686 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚493194340409344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚446083676608512 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22477665846272 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 208328050 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79999335 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚505519372260864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚505519348320768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚493194586629120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 208327394 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚977 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 219354686 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚471098747152896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22477938848768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 trainer/global_step‚11899 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚471098747152896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 219354684 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 309578004 YFDeviceStatsMonitor.on_train_batch_enú˜Ø^î d/allocated_bytes.small_pool.freed‚22095593256448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22095821473280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 528958386 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚471098765155840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 309603700 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22031692037193 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 528932688 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 309578004 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚26Þÿ¿'æ,úM B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚505519372260864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚505519348320768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚493194586629120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 208327394 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚977 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 219354686 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚471098747152896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22477938848768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273002496 trainer/global_step‚11899 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚471098747152896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 219354684 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 309578004 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22095593256448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22095821473280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 528958386 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚471098765155840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 309603700 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22031692037193 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 528932688 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 309578004 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 219354684 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 128328713 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22031468387837 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79998681 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337968128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 309603700 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 219354686 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚493194340409344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚446083676608512 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22477665846272 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 208328050 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79999335 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚495163207317504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚472977306412032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22185654699008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚507536687377408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚495162961111040 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚495163207317504 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 128836110 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507536663437312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307441664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚447862497043456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚11543.398235229 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 531056270 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚11949 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 209153268 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 310833355 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 128836108 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚472977324414976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 310833355 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚495162961111040 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚495163207317504 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 128836110 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507536663437312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 307441664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚447862497043456 E=DeviceStatsMonitor.on_traÏý‹M"in_batch_start/segment.all.allocated‚561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 _runtime‚11543.398235229 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 531056270 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚11949 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 209153268 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 310833355 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 128836108 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚472977324414976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 310833355 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚495163207317504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚472977306412032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22185654699008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Ô%§¾Ä4ö,/ _wandbruntime‚11544  _runtime‚11544ô¤™Mû&÷,õM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 220240634 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22187720610304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 220240636 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 531099540 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚495203136749568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚473015644356096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530148947591680 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚11949 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 209170153 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚11544.255147164 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 209169466 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚495203382969344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 531125238 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 128846465 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530149252922880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚262 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚473015662359040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚473015662359040 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22187720610304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22123325608401 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 310858906 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 220240636 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚473015662359040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚473015662359040 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 228216832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22187720610304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22123325608401 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 310858906 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 220240636 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 531099540 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚495203136749568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚473015644356096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530148947591680 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 281391104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚11949 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 209170153 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚11544.255147164 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 209169466 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚495203382969344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 531125238 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397ÕÂô×84ù,/ _wandbruntime‚11544  _runtime‚11544/»j/\ú,Ö 0loss/SRqqt4/detection_loss‚2.004593441883723 -loss/FBt1/detection_loss‚2.76813397804896 /loss/FBt2/detection_loss‚0.6797719718770513 0loss/FRt4/assignment_loss‚3.1208922618474717 1loss/FBt2/assignment_loss‚0.18437160239993722 0loss/SRqqt1/detection_loss‚2.552702946588397 &loss/total_loss‚42.693991877523196 /loss/FRt4/detection_loss‚2.4463006922832946 epoch‚9 $ _timestamp‚1.788428073506352e+09 _step‚982 .loss/FRt1/detection_loss‚2.559943463653326 0loss/FRt3/assignment_loss‚3.0853452667593957 2loss/SRqqt2/assignment_loss‚1.0551224091653226 2loss/SRqqt4/assignment_loss‚1.3530374167575723 .loss/FBt4/detection_loss‚2.328189422686895 /loss/FRt2/detection_loss‚1.4933634716652187 2loss/SRqqt1/assignment_loss‚1.6346637822862249 2loss/FBt3/assignment_loss‚0.008831114717759192 .loss/FRt3/detection_loss‚2.666003154218197 1loss/FBt1/assignment_loss‚0.20408317466171866 /loss/FBt3/detection_loss‚3.5352141499519347 _runtime‚11544.256175832 /loss/FRt2/assignment_loss‚1.565659409644557 trainer/global_step‚11949 /loss/FRt1/assignment_loss‚3.564519837498665 2loss/SRqqt3/assignment_loss‚1.3715750407062324 1loss/FBt4/assignment_loss‚0.03853312183249121 1loss/SRqqt2/detection_loss‚0.7613671954109598 0loss/SRqqt3/detection_loss‚1.711773550977894ÓÆœaû,Û 1loss/FBt1/assignment_loss‚0.20408317466171866 /loss/FBt3/detection_loss‚3.5352141499519347 _runtime‚11544.256175832 /loss/FRt2/assignment_loss‚1.565659409644557 trainer/global_step‚11949 /loss/FRt1/assignment_loss‚3.564519837498665 2loss/SRqqt3/assignment_loss‚1.3715750407062324 1loss/FBt4/assignment_loss‚0.03853312183249121 1loss/SRqqt2/detection_loss‚0.7613671954109598 0loss/SRqqt3/detection_loss‚1.711773550977894 0loss/SRqqt4/detection_loss‚2.004593441883723 -loss/FBt1/detection_loss‚2.76813397804896 /loss/FBt2/detection_loss‚0.6797719718770513 0loss/FRt4/assignment_loss‚3.1208922618474717 1loss/FBt2/assignment_loss‚0.18437160239993722 0loss/SRqqt1/detection_loss‚2.552702946588397 &loss/total_loss‚42.693991877523196 /loss/FRt4/detection_loss‚2.4463006922832946 epoch‚9 $ _timestamp‚1.788428073506352e+09 _step‚982 .loss/FRt1/detection_loss‚2.559943463653326 0loss/FRt3/assignment_loss‚3.0853452667593957 2loss/SRqqt2/assignment_loss‚1.0551224091653226 2loss/SRqqt4/assignment_loss‚1.3530374167575723 .loss/FBt4/detection_loss‚2.328189422686895 /loss/FRt2/detection_loss‚1.4933634716652187 2loss/SRqqt1/assignment_loss‚1.6346637822862249 2loss/FBt3/assignment_loss‚0.008831114717759192 .loss/FRt3/detection_loss‚2.666003154218197ֳ߸ü,‚(:… ¯þäÔ»Îz gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?^ÔÄý,‚(:ì ¯þäÔû¥×| disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0! proc.memory.rssMB‚ 2775.234375 disk.rbd1.out‚ 5.88916015625 disk.dm-1.in‚ 7.31640625 disk.rbd4.out‚0' disk.nvme0c0n1.out‚53934.2001953125 disk.sda1.in‚ 0.05078125) proc.memory.percent‚16.93868637084961 cpu‚25.284561307273616 proc.cpu.threads‚17 network.sent‚ 3.0308218e+07" disk.dm-0.out‚53934.2001953125 disk.rbd3.out‚0 network.recv‚ 4.095148e+06) disk./.usagePercent‚73.73837925480426 disk.sda3.in‚ 7.31640625 disk.nvme0n1.in‚5.3125% disk.nvme0n1.out‚53934.2001953125$ proc.memory.availableMB‚420.3125 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_™+˳percent‚97.43461608886719 disk.dm-0.in‚5.3125 disk.dm-1.out‚ 524.87109375 disk.rbd2.in‚0 disk.sda.out‚ 524.91015625 disk.rbd1.in‚0 disk.sda2.out‚ 0.0390625 disk.sda2.in‚0! disk.rbd2.out‚381.80712890625 disk.drbd1777.in‚ 1.03515625 disk.sda.in‚ 7.3671875% disk./.usageGB‚1098.7551383972168 disk.sda3.out‚ 524.87109375 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd4.in‚0îž›þ,‚(:… ¾þäÔê¼ôy gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.9*vuÿ,‚(:ê ¾þäÔ®…Ÿ| disk.sda1.out‚0 disk.nvme0n1.in‚5.3125$ memory_percent‚97.43759632110596 disk.sda3.out‚ 525.234375 disk.sda2.in‚0) proc.memory.percent‚16.93868637084961% disk.nvme0n1.out‚54033.3447265625% disk./.usageGB‚1098.7552604675293 disk.dm-1.out‚ 525.234375' disk.nvme0c0n1.out‚54033.3447265625 disk.rbd3.out‚0" disk.dm-0.out‚54033.3447265625 proc.cpu.threads‚17) disk./.usagePercent‚73.73838744704535 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.sda3.in‚ 7.31640625 cpu‚25.284259796736386 network.sent‚ 3.0417057e+07 network.recv‚ 4.10171e+06 disk.dm-1.in‚ 7.31640625 disk.rbd2.in‚0! disk.rbd2.out‚383.38134765625 disk.rbd3.in‚0 disk.sda.in‚ 7.3671875 disk.sda.out‚ 525.2734375 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚5.3125( proc.memory.availableMB‚ 419.82421875! proc.memory.rssMB‚ 2775.234375 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd1.out‚ 5.88916015625 disk.sda2.out‚ 0.0390625”Ù…=›€-‚(: ÍþäÔÔ½¤y gpu.0.gpu‚21.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.066' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.813200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´÷?;~-‚(:ó ÍþäԘ‘z disk.rbd0.in‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125$ memory_percent‚97.43630886077881 network.sent‚ 3.0423565e+07 disk.rbd1.out‚ 5.88916015625# disk.dm-0.out‚54132.47607421875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚54132.47607421875 disk.rbd3.out‚0) disk./.usagePercent‚73.73838744704535 disk.dm-1.out‚ 526.05078125 network.recv‚ 4.106723e+06 disk.sda.in‚ 7.3671875 cpu‚25.283945021173434 disk.nvme0n1.in‚5.3125& disk.nvme0n1.out‚54132.47607421875! disk.rbd2.out‚383.55322265625 disk.sda3.out‚ 526.05078125( proc.memory.availableMB‚ 420.03515625 proc.cpu.threads‚17% disk./.usageGB‚1098.7552604675293 disk.sda.out‚ 526.08984375! proc.memory.rssMB‚ 2775.234375) proc.memory.percent‚16.93868637084961 disk.dm-0.in‚5.3125 disk.dm-1.in‚ 7.31640625 disk.rbd2.in‚0 disk.sda1.in‚ 0.05078125 disk.sda3.in‚ 7.31640625 disk.rbd1.in‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625°À4‚-/ _wandbruntime‚11588  _runtime‚11588Î%µß˃-Å trainer/global_step‚11999 % _timestamp‚1.7884281181499162e+09 _runtime‚11588.899917395 _step‚983 ' lr-AdamW/pg1‚1.3845442288586687e-07 ' lr-AdamW/pg2‚1.3845442288586687e-07‡3ðoЄ-Ê % _timestamp‚1.7884281181499162e+09 _runtime‚11588.899917395 _step‚983 ' lr-AdamW/pg1‚1.3845442288586687e-07 ' lr-AdamW/pg2‚1.3845442288586687e-07 trainer/global_step‚11999×+›Û4…-/ _wandbruntime‚11588  _runtime‚11588éíÞ¨ñ'†-ëO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚474894221618176 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22662988075520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6252512256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚532257857018368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚11999 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22212903130653 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497171757451264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚474894203615232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚471890523066909 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449677636975616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22213126779801 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 312114257 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 312139927 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22277782039552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129353860 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚509595172190208 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 209992783 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 _runtime‚11588.900847437 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497171757451264 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 221108867 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚497172003657728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497172003657728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 221108867 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚474894203615232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _step‚984 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 209992120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 221108865 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚474894221618176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22277553836032 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22277553836032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚532258160265728 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚509595148250112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129353858 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚471890763755417 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884281181506894e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 533248794 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 312114257 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 533223122 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 221108865 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80638923 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22662708768256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497171757451264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚474894203615232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚471890523066909 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449677636975616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22213126779801 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 312114257 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 312139927 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 223649148 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22277782039552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129353860 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1261 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚509595172190208 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 209992783 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 303247360 _runtime‚11588.900847437 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 549453824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497171757451264 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 221108867 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚497172003657728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 279307264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚561 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497172003657728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 221108867 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚474894203615232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚262 _step‚984 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 209992120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚23 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 221108865 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 246206464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6039636992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚474894221618176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 549453824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22277553836032 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22277553836032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚532258160265728 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚509595148250112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129353858 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚471890763755417 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884281181506894e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 533248794 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 312114257 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 533223122 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 221108865 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80638923 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22662708768256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 240688508 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210008779 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497212179309568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚532300889502208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚11999 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 221126586 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 312139808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚474932541559296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚532301192736256 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚471928898613081 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 533292090 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497211933089792 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129364215 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129364213 $ _timestamp‚1.788428119020151e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22279391530496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22214959179609 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22279619747328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 533266392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚474932559562240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 533266392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22214735530253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚474932559562240 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 312165504 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 221126586 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 210009456 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449713939433472 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 533292090 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚675 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22664571555840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚474932541559296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80644566 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/actiüÚhËDve.large_pool.freed‚ 221126584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80645241 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚509636317946368 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 312139808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 549453824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210008779 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497212179309568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚532300889502208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚11999 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚23 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 221126586 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 312139808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚474932541559296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚532301192736256 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 303234048 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚471928898613081 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 533292090 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497211933089792 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129364215 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129364213 $ _timestamp‚1.788428119020151e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22279391530496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22214959179609 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22279619747328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 533266392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 279293952 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚474932559562240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 533266392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22214735530253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚474932559562240 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 312165504 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 221126586 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 210009456 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449713939433472 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 533292090 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚675 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22664571555840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚474932541559296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80644566 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 221126584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 541892608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80645241 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2ÙW4‹-/ _wandbruntime‚11589  _runtime‚11589]`á­ZŒ-Ô 0loss/FRt2/assignment_loss‚1.2803153750744272 0loss/FRt4/assignment_loss‚3.4601223184303804 2loss/SRqqt2/assignment_loss‚0.6695558596402407 0loss/FBt1/assignment_loss‚0.2485537452303106 1loss/FBt2/assignment_loss‚0.09900371706325105 .loss/FRt1/detection_loss‚2.593607230732838 /loss/FRt4/detection_loss‚2.1480676587094085 &loss/total_loss‚39.149406630398424 _runtime‚11589.771667777 -loss/FRt3/detection_loss‚2.42251718044281 1loss/SRqqt2/detection_loss‚0.7279459255348358 .loss/FBt1/detection_loss‚2.479543261229992 -loss/FBt4/detection_loss‚2.77991361618042 % _timestamp‚1.7884281190218248e+09 0loss/FRt1/assignment_loss‚2.7179500742091074 /loss/FRt3/assignment_loss‚2.580787569284439 2loss/SRqqt4/assignment_loss‚1.4434561426192523 trainer/global_step‚11999 /loss/FBt2/detection_loss‚0.5609942894648103 .loss/FBt3/detection_loss‚2.207134835422039 0loss/SRqqt1/detection_loss‚2.548558616389831 _step‚986 1loss/FBt4/assignment_loss‚0.18842107011005282 1loss/SRqqt3/detection_loss‚1.7503203675150871 epoch‚9 1loss/SRqqt1/assignment_loss‚1.484935642258885 1loss/SRqqt3/assignment_loss‚1.410565357694092 1loss/FBt3/assignment_loss‚0.12142129070707597 /loss/FRt2/detection_loss‚1.4328963578222913 1loss/SRqqt4/detection_loss‚1.7928191286325454õÎôõ_-Ù 1loss/SRqqt3/assignment_loss‚1.410565357694092 1loss/FBt3/assignment_loss‚0.12142129070707597 /loss/FRt2/detection_loss‚1.4328963578222913 1loss/SRqqt4/detection_loss‚1.7928191286325454 0loss/FRt2/assignment_loss‚1.2803153750744272 0loss/FRt4/assignment_loss‚3.4601223184303804 2loss/SRqqt2/assignment_loss‚0.6695558596402407 0loss/FBt1/assignment_loss‚0.2485537452303106 1loss/FBt2/assignment_loss‚0.09900371706325105 .loss/FRt1/detection_loss‚2.593607230732838 /loss/FRt4/detection_loss‚2.1480676587094085 &loss/total_loss‚39.149406630398424 _runtime‚11589.771667777 -loss/FRt3/detection_loss‚2.42251718044281 1loss/SRqqt2/detection_loss‚0.7279459255348358 .loss/FBt1/detection_loss‚2.479543261229992 -loss/FBt4/detection_loss‚2.77991361618042 % _timestamp‚1.7884281190218248e+09 0loss/FRt1/assignment_loss‚2.7179500742091074 /loss/FRt3/assignment_loss‚2.580787569284439 2loss/SRqqt4/assignment_loss‚1.4434561426192523 trainer/global_step‚11999 /loss/FBt2/detection_loss‚0.5609942894648103 .loss/FBt3/detection_loss‚2.207134835422039 0loss/SRqqt1/detection_loss‚2.548558616389831 _step‚986 1loss/FBt4/assignment_loss‚0.18842107011005282 1loss/SRqqt3/detection_loss‚1.7503203675150871 epoch‚9 1loss/SRqqt1/assignment_loss‚1.484935642258885ÚcgëÄŽ-‚(:… ÜþäÔ¹•½x gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.896' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wÞÿ½u-‚(:ê Üþäԃ׉|( disk.nvme0c0n1.out‚54165.16357421875 disk.rbd3.out‚0 network.recv‚ 4.110103e+06! disk.rbd2.out‚385.12744140625 disk.rbd4.in‚0( proc.memory.availableMB‚ 420.15234375% disk./.usageGB‚1098.7552680969238& disk.nvme0n1.out‚54165.16357421875 disk.sda2.out‚ 0.0390625 disk.dm-1.in‚ 7.31640625 disk.dm-1.out‚ 526.78125 disk.sda.out‚ 526.8203125 disk.sda3.in‚ 7.31640625 cpu‚25.28365581221977 disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.sda1.out‚0 disk.sda3.out‚ 526.78125 disk.rbd1.in‚0# memory_percent‚97.4355936050415! proc.memory.rssMB‚ 2775.234375# disk.dm-0.out‚54149.28857421875 disk.drbd1777.out‚0 disk.sda1.in‚ 0.05078125 proc.cpu.threads‚17 network.sent‚ 3.0428814e+07) disk./.usagePercent‚73.73838795906042 disk.dm-0.in‚5.3125 disk.rbd0.in‚0 disk.rbd4.out‚0 disk.rbd1.out‚ 5.88916015625 disk.sda2.in‚0) proc.memory.percent‚16.93868637084961 disk.rbd2.in‚0 disk.nvme0n1.in‚5.3125 disk.nvme0c0n1.in‚5.3125 disk.rbd3.in‚0 disk.sda.in‚ 7.3671875M¢Þ&-‚(:„ ëþäԔݖx gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ça’Áx‘-‚(:í ëþäÔ’Òäz disk.sda1.out‚0 disk.dm-1.in‚ 7.31640625 disk.dm-1.out‚ 527.25390625 disk.rbd4.out‚0 disk.nvme0c0n1.in‚5.3125 disk.sda3.in‚ 7.31640625 disk.sda3.out‚ 527.25390625 disk.rbd1.out‚ 5.88916015625$ memory_percent‚97.43423461914062 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd4.in‚0 disk.sda.in‚ 7.3671875 disk.sda2.in‚0 cpu‚25.28333894041073 network.recv‚ 4.116162e+06 disk.rbd3.in‚0) proc.memory.percent‚16.93868637084961 disk.drbd1777.in‚ 1.03515625 disk.sda1.in‚ 0.05078125 disk.nvme0n1.in‚5.3125# proc.memory.availableMB‚420.375! proc.memory.rssMB‚ 2775.234375 disk.dm-0.in‚5.3125# disk.dm-0.out‚54232.45458984375 disk.rbd0.out‚0% disk./.usageGB‚1098.7544441223145( disk.nvme0c0n1.out‚54232.45458984375 disk.rbd3.out‚0& disk.nvme0n1.out‚54232.45458984375 disk.sda2.out‚ 0.0390625! disk.rbd2.out‚385.23291015625 network.sent‚ 3.0537473e+07) disk./.usagePercent‚73.73833266143302 disk.sda.out‚ 527.29296875 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0š8Ù¦š’-‚(: úþäÔ÷€Ôx gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.46' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.491999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´QKÔz“-‚(:ï úþäÔÄÍ„{ disk.rbd0.out‚0 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125 disk.nvme0n1.in‚5.3125 disk.dm-1.out‚ 527.6640625 disk.rbd4.in‚0 disk.sda.out‚ 527.703125 network.sent‚ 3.0546964e+07) disk./.usagePercent‚73.73833266143302 disk.sda2.out‚ 0.0390625 disk.sda1.in‚ 0.05078125 disk.dm-0.in‚5.3125! disk.rbd2.out‚386.80712890625 disk.sda1.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 527.6640625 disk.rbd1.out‚ 5.88916015625& disk.nvme0n1.out‚54331.50537109375 disk.dm-1.in‚ 7.31640625 disk.rbd2.in‚0 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 7.3671875 network.recv‚ 4.127128e+06( disk.nvme0c0n1.out‚54331.50537109375! proc.memory.rssMB‚ 2775.234375 cpu‚25.283037400579822% disk./.usageGB‚1098.7544441223145 disk.sda3.in‚ 7.31640625 disk.rbd1.in‚0$ memory_percent‚97.43425846099854 disk.sda2.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 420.37109375) proc.memory.percent‚16.93868637084961# disk.dm-0.out‚54331.50537109375Jt€-4”-/ _wandbruntime‚11634  _runtime‚11634öÅÝNË•-Å _step‚987 ' lr-AdamW/pg1‚2.8106817539913147e-08 ' lr-AdamW/pg2‚2.8106817539913147e-08 trainer/global_step‚12049 % _timestamp‚1.7884281638323517e+09 _runtime‚11634.582251181¸ŸÇ0Ж-Ê _step‚987 ' lr-AdamW/pg1‚2.8106817539913147e-08 ' lr-AdamW/pg2‚2.8106817539913147e-08 trainer/global_step‚12049 % _timestamp‚1.7884281638323517e+09 _runtime‚11634.582251181Û¾f²¨4—-/ _wandbruntime‚11634  _runtime‚11634rý—ð'˜-êO `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22756170046464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚561 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 313395159 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚473797279530917 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 535389974 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129871608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚499180553791488 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 313420829 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 535415646 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 trainer/global_step‚12049 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 313395159 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129871610 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚476811100818432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 313420829 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚659 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚499180799997952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚511653633062912 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚476811118821376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚499180799997952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚511653657003008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22369452973056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476811100818432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 210832213 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚534410100065280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚499180553791488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚451492742829056 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22369681176576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚25672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31397 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚473797279530917 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚561 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 535389974 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6739344896 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚90898432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚262 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31168 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129871608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚561 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚499180553791488 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 313420829 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 228203520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 535415646 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1243 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6343884800 trainer/global_step‚12049 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 313395159 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6739344896 YKDeviceStatsMonitor&Küä¯.on_train_batch_start/inactive_split.large_pool.allocated‚ 129871610 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31397 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚476811100818432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 313420829 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6893338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 246206464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚25670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚659 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚499180799997952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚511653633062912 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537294673 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚476811118821376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 541892608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚499180799997952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337968128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚511653657003008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6893338624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22369452973056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚262 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476811100818432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6252512256 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 210832213 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚534410100065280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚499180553791488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚25672 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚25670 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚451492742829056 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 541892608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3428 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22369681176576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31168 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6522528073 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 228203520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3428 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210848670 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22306592750817 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22306369101461 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 535433244 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129881963 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚473835655077089 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚511694802759168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476849438762496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 222012534 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22371518884352 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚499220975649792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚451529062326272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚12049 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚476849438762496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚499220729430016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 535433244 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80966707 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 210849276 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚476849456765440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 313420710 % _timestamp‚1.7884281647072358e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 535458942 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80967311 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22758307802624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 222012536 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 313446406 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 313446406 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129881965 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚451529045286912 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22371290667520 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚499220975649792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22758036897280 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476849456765440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22371290667520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 535458942 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 _runtime‚11635.457690722 _step‚989 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22371518884352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 222012534 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚473835414388373 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚25698 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210848670 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22306592750817 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6893338624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚262 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚561 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22306369101461 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚561 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚299 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 246219776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 535433244 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129881963 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 246219776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6739344896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚473835655077089 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6739344896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3428 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚511694802759168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6893338624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6252512256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476849438762496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 222012534 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22371518884352 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚499220975649792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 223649356 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537294673 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚561 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6522528073 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚451529062326272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚25696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 541892608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚12049 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚476849438762496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚499220729430016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 535433244 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80966707 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6343884800 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 210849276 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1261 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚299 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 549453824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚476849456765440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 313420710 % _timestamp‚1.7884281647072358e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 535458942 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31397 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80967311 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22758307802624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 222012536 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 313446406 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6039636992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 313446406 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31168 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129881965 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 240688716 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚451529045286912 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6893338624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 549453824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 309910528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22371290667520 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚499220975649792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270905344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22758036897280 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476849456765440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22371290667520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31397 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 535458942 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚262 _runtime‚11635.457690722 _step‚989 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 228216832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚90898432 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚25696 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22371518884352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 222012534 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1243 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294845440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0Ý ·!è4-/ _wandbruntime‚11635  _runtime‚11635ó=ä÷až-Û 0loss/FRt4/assignment_loss‚2.9257963364538937 .loss/FRt4/detection_loss‚2.360420118207517 1loss/SRqqt4/detection_loss‚1.6882092898986378 2loss/SRqqt2/assignment_loss‚0.6826709034357918 1loss/FBt4/assignment_loss‚0.09215707719946901 /loss/FRt1/detection_loss‚2.5139920411631467 0loss/SRqqt1/detection_loss‚2.297541587641745 0loss/SRqqt2/detection_loss‚0.754292172845453 /loss/FBt3/detection_loss‚2.4980220913887026 epoch‚9 trainer/global_step‚12049 0loss/FRt2/assignment_loss‚1.2269580134978662 1loss/FBt2/assignment_loss‚0.12040215650792899 2loss/FBt3/assignment_loss‚0.006943997158668935 % _timestamp‚1.7884281647089248e+09 /loss/FRt1/assignment_loss‚3.154942384362221 1loss/SRqqt3/detection_loss‚1.7166881278157233 /loss/FBt1/detection_loss‚1.9187191128730774 2loss/SRqqt1/assignment_loss‚1.5207424583760174 .loss/FRt2/detection_loss‚1.789394030204186 /loss/FBt4/detection_loss‚2.5209377308686576 %loss/total_loss‚39.20056310178654 _step‚990 /loss/FRt3/assignment_loss‚3.425010163675655 2loss/SRqqt3/assignment_loss‚1.4632545259005079 0loss/FBt2/detection_loss‚0.45874072792181786 2loss/SRqqt4/assignment_loss‚1.2614618366370436 1loss/FBt1/assignment_loss‚0.22675491258860347 /loss/FRt3/detection_loss‚2.5765113051642072 _runtime‚11635.458739218"ЙfŸ-à /loss/FRt1/assignment_loss‚3.154942384362221 1loss/SRqqt3/detection_loss‚1.7166881278157233 /loss/FBt1/detection_loss‚1.9187191128730774 2loss/SRqqt1/assignment_loss‚1.5207424583760174 .loss/FRt2/detection_loss‚1.789394030204186 /loss/FBt4/detection_loss‚2.5209377308686576 %loss/total_loss‚39.20056310178654 _step‚990 /loss/FRt3/assignment_loss‚3.425010163675655 2loss/SRqqt3/assignment_loss‚1.4632545259005079 0loss/FBt2/detection_loss‚0.45874072792181786 2loss/SRqqt4/assignment_loss‚1.2614618366370436 1loss/FBt1/assignment_loss‚0.22675491258860347 /loss/FRt3/detection_loss‚2.5765113051642072 _runtime‚11635.458739218 0loss/FRt4/assignment_loss‚2.9257963364538937 .loss/FRt4/detection_loss‚2.360420118207517 1loss/SRqqt4/detection_loss‚1.6882092898986378 2loss/SRqqt2/assignment_loss‚0.6826709034357918 1loss/FBt4/assignment_loss‚0.09215707719946901 /loss/FRt1/detection_loss‚2.5139920411631467 0loss/SRqqt1/detection_loss‚2.297541587641745 0loss/SRqqt2/detection_loss‚0.754292172845453 /loss/FBt3/detection_loss‚2.4980220913887026 epoch‚9 trainer/global_step‚12049 0loss/FRt2/assignment_loss‚1.2269580134978662 1loss/FBt2/assignment_loss‚0.12040215650792899 2loss/FBt3/assignment_loss‚0.006943997158668935 % _timestamp‚1.7884281647089248e+09Þ3æƒÓ -‚(:… ‰ÿäÔ½ÿ x gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.129' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6258 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Ƀw¡-‚(:ì ‰ÿäÔþá•| cpu‚25.28273275810802 disk.sda3.in‚ 7.31640625 disk.rbd1.out‚ 5.88916015625# disk.dm-0.out‚54430.79248046875 disk.dm-1.in‚ 7.31640625% proc.memory.availableMB‚ 419.84375 disk.rbd0.out‚0 disk.sda1.in‚ 0.05078125) proc.memory.percent‚16.93868637084961 disk.rbd2.in‚0( disk.nvme0c0n1.out‚54430.79248046875 disk.rbd3.out‚0 disk.sda.in‚ 7.3671875 disk.sda3.out‚ 528.39453125 disk.nvme0c0n1.in‚5.3125 disk.rbd1.in‚0& disk.nvme0n1.out‚54430.79248046875# disk./.usageGB‚1098.75541305542 disk.rbd0.in‚0 disk.sda.out‚ 528.43359375 disk.nvme0n1.in‚5.3125 proc.cpu.threads‚17) disk./.usagePercent‚73.73839768734672 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.dm-0.in‚5.3125 disk.sda2.out‚ 0.0390625 disk.dm-1.out‚ 528.39453125! proc.memory.rssMB‚ 2775.234375 network.recv‚ 4.131995e+06 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 network.sent‚ 3.0553265e+07! disk.rbd2.out‚386.97900390625 disk.drbd1777.in‚ 1.03515625 disk.drbd1777.out‚0# memory_percent‚97.4374771118164൵¢-‚(:‚ ˜ÿäÔÒÅšx gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚90.32' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.064 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h¦Ø9F£-‚(:î ˜ÿäÔˆŒ²z$ memory_percent‚97.43449687957764 disk.rbd1.in‚0 disk.nvme0c0n1.in‚5.3125 cpu‚25.282420437608785 proc.cpu.threads‚17 disk.sda3.in‚ 7.31640625 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.sda3.out‚ 528.8671875! disk.rbd2.out‚388.55322265625 disk.sda.in‚ 7.oès33671875) disk./.usagePercent‚73.73839640730905 disk.rbd1.out‚ 5.88916015625 disk.dm-1.in‚ 7.31640625 disk.sda1.out‚0% disk./.usageGB‚1098.7553939819336 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0 disk.rbd4.out‚0 disk.sda.out‚ 528.90625( proc.memory.availableMB‚ 420.33203125& disk.nvme0n1.out‚54530.06298828125 disk.dm-0.in‚5.3125 disk.rbd4.in‚0 disk.rbd3.in‚0) proc.memory.percent‚16.93868637084961 network.sent‚ 3.0661429e+07# disk.dm-0.out‚54530.06298828125 disk.sda2.out‚ 0.0390625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚ 1.03515625! proc.memory.rssMB‚ 2775.234375 network.recv‚ 4.137547e+06 disk.dm-1.out‚ 528.8671875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚54530.06298828125 disk.sda1.in‚ 0.05078125ClnD¤-‚(:… §ÿäÔ¡ž°x gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.073' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.6146 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0us¥Ír¥-‚(:ç §ÿäÔ‹‹Ùz disk.rbd1.in‚0 disk.dm-0.out‚54629.39453125 disk.rbd0.in‚0 disk.sda3.out‚ 529.68359375 disk.sda2.out‚ 0.0390625' proc.memory.availableMB‚ 420.3046875 network.sent‚ 3.066647e+07 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.sda1.in‚ 0.05078125 disk.sda.out‚ 529.72265625 disk.dm-0.in‚5.3125 disk.sda2.in‚0 disk.dm-1.in‚ 7.31640625% disk./.usageGB‚1098.7553100585938 disk.sda.in‚ 7.3671875! disk.rbd2.out‚390.29541015625) proc.memory.percent‚16.93868637084961 proc.cpu.threads‚17 disk.drbd1777.in‚ 1.03515625 disk.rbd4.in‚0 disk.dm-1.out‚ 529.68359375! proc.memory.rssMB‚ 2775.234375 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 5.88916015625 disk.nvme0n1.in‚5.3125 disk.nvme0c0n1.in‚5.3125 disk.sda3.in‚ 7.31640625 disk.rbd2.in‚0( disk./.usagePercent‚73.7383907751433 disk.rbd3.in‚0# disk.nvme0n1.out‚54629.39453125$ memory_percent‚97.43466377258301 cpu‚25.282121016774738 network.recv‚ 4.141319e+06% disk.nvme0c0n1.out‚54629.39453125 disk.rbd3.out‚0s(mŽ¦-‚(:ƒ ¶ÿäÔ÷õåx gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.424' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iÀ {§-‚(:ð ¶ÿäÔ÷»¡{" disk.dm-0.out‚54630.0869140625 proc.cpu.threads‚17 disk.dm-1.out‚ 530.67578125 disk.drbd1777.in‚ 1.03515625 disk.rbd4.out‚0 disk.sda.in‚ 7.3671875 network.sent‚ 3.0672462e+07 disk.sda2.in‚0 disk.dm-1.in‚ 7.31640625 disk.sda3.in‚ 7.31640625( proc.memory.availableMB‚ 343.80078125) proc.memory.percent‚17.40405559539795! disk.rbd2.out‚390.29541015625 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 530.71484375 disk.nvme0n1.in‚5.3125% disk.nvme0n1.out‚54630.0869140625 disk.dm-0.in‚5.3125 disk.rbd3.out‚0 cpu‚25.28673086106851 disk.rbd2.in‚0 disk.nvme0c0n1.in‚5.3125 disk.rbd1.in‚0 disk.rbd1.out‚ 5.88916015625# proc.memory.rssMB‚ 2851.48046875 disk.sda3.out‚ 530.67578125) disk./.usagePercent‚73.73837567069877 disk.rbd0.in‚0 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚54630.0869140625 disk.sda1.out‚0$ memory_percent‚97.90160655975342 network.recv‚ 4.145421e+06 disk.sda1.in‚ 0.05078125$ disk./.usageGB‚1098.755084991455 disk.sda2.out‚ 0.03906259E쎨-‚(:ƒ ÅÿäÔŸ¯´x gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.424' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Hó½Ìr©-‚(:ç ÅÿäÔÔÎÿz disk.rbd4.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.in‚ 7.31640625 disk.dm-1.out‚ 531.48828125 disk.rbd0.out‚0" proc.memory.rssMB‚ 2851.7421875) proc.memory.percent‚17.40565299987793 disk.nvme0n1.in‚5.3125 disk.dm-1.in‚ 7.31640625 disk.rbd4.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 network.recv‚ 4.150459e+06% disk./.usageGB‚1098.7550659179688" disk.nvme0n1.out‚ 54729.7734375 disk.dm-0.in‚5.3125 disk.sda.in‚ 7.3671875 disk.sda.out‚ 531.52734375 cpu‚25.29252655220049 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚5.3125) disk./.usagePercent‚73.73837439066111 disk.sda2.in‚0 disk.sda2.out‚ 0.0390625! disk.rbd2.out‚391.97900390625 disk.drbd1777.in‚ 1.03515625 disk.rbd3.out‚0 network.sent‚ 3.0678803e+07 disk.rbd1.out‚ 5.88916015625 disk.dm-0.out‚ 54729.7734375 disk.sda1.in‚ 0.05078125( proc.memory.availableMB‚ 343.44921875 disk.sda3.out‚ 531.48828125 disk.rbd2.in‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 54729.7734375$ memory_percent‚97.90375232696533ûÔVŽª-‚(:ƒ ÔÿäÔ°¥x gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.424' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;ùÝv«-‚(:ë ÔÿäÔ»æÌ{ disk.rbd3.out‚0 disk.rbd1.out‚ 5.88916015625 disk.dm-0.out‚54828.88671875 disk.rbd2.in‚0 cpu‚25.298133265305136 disk.sda3.in‚ 7.31640625 disk.nvme0n1.in‚5.3125 disk.sda2.in‚0) proc.memory.percent‚17.40565299987793 network.recv‚ 4.153469e+06 disk.nvme0c0n1.in‚5.3125 disk.dm-1.out‚ 531.89453125% disk./.usageGB‚1098.7551231384277 disk.dm-0.in‚5.3125 disk.dm-1.in‚ 7.31640625" proc.memory.rssMB‚ 2851.7421875 network.sent‚ 3.0684083e+07% disk.nvme0c0n1.out‚54828.88671875 disk.sda3.out‚ 531.89453125 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.sda1.in‚ 0.05078125 disk.sda.in‚ 7.3671875 disk.rbd1.in‚0# disk.nvme0n1.out‚54828.88671875 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1777.out‚0$ memory_percent‚97.90375232696533) disk./.usagePercent‚73.73837823077413 disk.rbd4.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 531.93359375 disk.drbd1777.in‚ 1.03515625( proc.memory.availableMB‚ 343.44921875! disk.rbd2.out‚391.97900390625 disk.rbd4.in‚0ø \`4¬-/ _wandbruntime‚11724  _runtime‚11724¼JáX*­-ÒT KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚479663873226240 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3428 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 314875288 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚476592951132325 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 171710976 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚537537096524800 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚262 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚454194038114304 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚514684925211136 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31168 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6739344896 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚262 trainer/global_step‚12089 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 538308245 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚23 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 189713920 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚22464125444096 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚479663873226240 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 538328448 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚561 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 223453160 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 212222160 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚262 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚22464125444096 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337968128 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚561 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚479663855223296 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 171710976 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 541892608 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚81486759 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚22464297155072 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚22464297155072 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 6893338624 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6343884800 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 223453158 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚514684949151232 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20201 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _step‚991H§ +]*®-×T VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 223453160 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 549453824 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3428 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20201 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚81486391 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6522528073 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚502128170381312 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1261 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚22399080839421 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚502127980667392 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1243 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚476593135993085 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 6893338624 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚502128170381312 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31397 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚90898432 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20203 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚479663855223296 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚22852171313664 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚299 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚502127980667392 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 179384832 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 537294673 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6343884800 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 223453158 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 314855087 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 314855087 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚22464297155072 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚22464297155072 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 6893338624 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6343884800 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 223453158 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚514684949151232 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20201 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6039636992 _step‚991 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 549453824 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚479663873226240 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3428 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 314875288 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚476592951132325 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 171710976 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚537537096524800 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚561 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚262 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚454194038114304 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚514684925211136 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31168 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6739344896 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 309910528 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚262 trainer/global_step‚12089 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 538308245 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚23 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 189713920 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚22464125444096 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚479663873226240 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 538328448 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 549453824 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚561 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 541892608 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 223453160 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 212222160 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚262 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚22464125444096 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337968128 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6343884800 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚561 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6739344896 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚479663855223296 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 171710976 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 541892608 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚81486759 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 223468094 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3428 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚479689996013056 \ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚90898432 _step‚992 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚23 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚22399472016317 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 538351660 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚514712340203008 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚262 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31397 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚476618280119229 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚22399304194917 _runtime‚11725.954041814 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 223468094À‡‹nf)±-àR UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚502154514252288 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 130744363 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚537564904311808 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20203 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31397 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31168 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚537565083696640 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337968128 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 212236062 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚561 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚454218808102912 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6252512256 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 167821400 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 130744365 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6343884800 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚561 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 223468094 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3428 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6343884800 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚479689996013056 \ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚90898432 _step‚992 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚23 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚22399472016317 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6252512256 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 538351660 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚514712340203008 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚262 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚262 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31397 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚476618280119229 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚22399304194917 _runtime‚11725.954041814 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 223468094à?;ä²-‚(:„ ãÿäÔ¯¢“x gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.664627075195312) gpu.0.memoryAllocatedBytes‚ 7442857984 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í îl³-‚(:á ãÿäÔÏœÐ{ disk.rbd0.out‚0' proc.memory.availableMB‚ 343.4609375 disk.drbd1777.in‚ 1.03515625% disk.nvme0c0n1.out‚54928.80078125 disk.sda.out‚ 532.4453125) disk./.usagePercent‚73.73837823077413 disk.nvme0c0n1.in‚5.3125 disk.sda2.in‚0 network.recv‚ 4.157922e+06 disk.sda1.in‚ 0.05078125 disk.rbd1.out‚ 5.88916015625 disk.dm-0.in‚5.3125! disk.rbd2.out‚393.74462890625# memory_percent‚97.9036808013916 cpu‚25.30381324381847 disk.rbd1.in‚0) proc.memory.percent‚17.40565299987793 disk.sda3.out‚ 532.40625 disk.dm-1.out‚ 532.40625" proc.memory.rssMB‚ 2851.7421875 network.sent‚ 3.0690142e+07 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 7.3671875 disk.dm-0.out‚54928.80078125 disk.sda2.out‚ 0.0390625 disk.rbd0.in‚0 disk.rbd4.in‚0# disk.nvme0n1.out‚54928.80078125 disk.dm-1.in‚ 7.31640625 disk.rbd2.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1098.7551231384277 disk.rbd4.out‚0 disk.rbd3.out‚0 disk.sda3.in‚ 7.31640625 disk.nvme0n1.in‚5.3125±¹J4´-/ _wandbruntime‚11739  _runtime‚11739€ÛÚ6µ-±§ ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.007258064113557339 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5309696793556213 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17730192840099335 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818182468414307 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42795565724372864 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9954648613929749 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9139065146446228 -jet/accuracy_4_of_5‚0.0070340498350560665 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13456284999847412 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.45900219678878784 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4476902484893799 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02202066034078598 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1513010710477829 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.14166666567325592 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4097001850605011 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.530308723449707 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3447779715061188 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1937241107225418 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03928571566939354 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10615752637386322 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6459094285964966 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17631317675113678 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909092128276825 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2518450915813446 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.46479716897010803 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5586805939674377 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3888888955116272 particle/accuracy_3_of_7‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3944444954395294 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_5‚0.4887993037700653 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39836594462394714 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1311294287443161 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 jet/accuracy_2_of_7‚0 jet/accuracy_6_of_7‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6842105388641357 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10823754966259003 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982219338417053 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 jet/accuracy_1_of_6‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5394098162651062 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4276094138622284 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08093836158514023 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17002823948860168 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_3_of_5‚1 -"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5625 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3860046863555908 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4493964910507202 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 *loss/val_total_loss‚41.214762346511144 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2716699242591858 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1643274873495102 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4561403691768646 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 'particle/accuracy‚0.853288471698761 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16997934877872467 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23116064071655273 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5155717730522156 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5239859223365784 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1736111044883728 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.32296502590179443 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13617977499961853 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 +jet/accuracy_3_of_3‚0.01917058788239956 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5472807884216309 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666269302368 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_6‚1 jet/accuracy_1_of_7‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.022440627217292786 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2961314618587494 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16145820915699005 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20459268987178802 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4022029936313629 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.494047611951828 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 +particle/sensitivity‚0.3240649402141571 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.40333330631256104 particle/accuracy_1_of_7‚1 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16145820915699005 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17875371873378754 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.027411900460720062 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21494251489639282 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 epoch‚9 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.822483241558075 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4954071342945099 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14453798532485962 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4734683632850647 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.011437909677624702 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.13333334028720856 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.44516077637672424 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9930745363235474 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9938411116600037 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38226962089538574 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.443513423204422 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13517972826957703 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.20580807328224182 +jet/accuracy_2_of_2‚0.10414063185453415 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0644872859120369 particle/accuracy_1_of_3‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975067377090454 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4046783149242401 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1967547982931137 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39263394474983215 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40275508165359497 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.007258064113557339 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 particle/accuracy_6_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0674603208899498 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4607204496860504 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.15625 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12764334678649902 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/0FRt2SRqqt3÷tš™]FBt/SRqqt_purity‚0.5 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9855191111564636 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007407407276332378 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.042635660618543625 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968628287315369 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.15625 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06028369069099426 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5122374892234802 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.060452066361904144 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9930830597877502 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4523819088935852 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.71875 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.413194477558136 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12522676587104797 *jet/accuracy_2_of_4‚0.3583986163139343 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.05781489983201027 jet/accuracy_5_of_6‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.41929295659065247 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.24396973848342896 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.26897308230400085 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46655434370040894 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4369369149208069 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 trainer/global_step‚12089 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0072058457881212234 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5253464579582214 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.42548272013664246 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4127906858921051 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6755685210227966 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14526574313640594 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07381182163953781 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13755717873573303 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.13461539149284363 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.36970648169517517 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_6‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19862817227840424 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3433035910129547 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754689216614 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5666666626930237 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.026164688169956207 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 6 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.123795285820961 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.533699631690979 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2925925850868225 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013310089707374573 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_4_of_4‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9903714656829834 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12494172155857086 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_5_of_6‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5399700403213501 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5833231210708618 *jet/accuracy_1_of_3‚0.8529576063156128 *jet/accuracy_1_of_5‚0.9300691485404968 +jet/accuracy_2_of_6‚0.47435900568962097 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24237830936908722 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4406779408454895 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_3‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9994708895683289 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.7222222089767456 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4195401966571808 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.39945533871650696 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3890621066093445 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9845558404922485 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10133533924818039 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.46462005376815796 'particle/f_score‚0.4080885648727417 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38006988167762756 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.46206894516944885 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123459696769714 _runtime‚11739.907292801 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14451533555984497 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.019396550953388214 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43183085322380066 ) Purity/3FRt1SRqqt1FBt/FRt_purity‚0.25 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.011437909677624702 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9911864399909973 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5488251447677612 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_5_of_7‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10909444838762283 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016875647008419037 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7355797290802002 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5031671524047852 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08662201464176178 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4834437966346741 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4173569083213806 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22836972773075104 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.47877761721611023 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6764705777168274 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5049228072166443 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1505754142999649 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18918919563293457 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 particle/accuracy_2_of_7‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9914783239364624 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_7‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.31274688243865967 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9910677671432495 jet/accuracy_6_of_6‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.513451874256134 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.520516574382782 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5603677034378052 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4480288028717041 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2021058201789856 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 6validation_average_jet_accuracy‚0.4215868413448334 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9733333587646484 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.015809010714292526 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14647655189037323 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3757380247116089 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.253352552652359 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 particle/accuracy_2_of_5‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_1_of_4‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4799100458621979 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.477710485458374 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.36666667461395264 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9512432217597961 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14626437425613403 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.021813547238707542 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4600282609462738 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.23406751453876495 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9928160905838013 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5583333373069763 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09555555135011673 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.06578947603702545 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 _step‚993 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 particle/accuracy_7_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9886921644210815 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.013701708056032658 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002365616150200367 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24438293278217316 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1688058227300644 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9922755360603333 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.44948869943618774 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.49790191650390625 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 5 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.49038165807724 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12037035822868347 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16284695267677307 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00805696565657854 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2311508059501648 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4600282609462738 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9952381253242493 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2490740865468979 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7884282691550264e+09 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 particle/accuracy_3_of_4‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9904405474662781 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.48779600858688354 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05480317398905754 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5029312968254089 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.09935609996318817 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14022952318191528 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.005201044492423534 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20276951789855957 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.992125391960144 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4880274534225464 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3901515305042267 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5251725316047668 jet/accuracy_7_of_7‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23736511170864105 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 jet/accuracy_4_of_7‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125¥¬G©Y"¶-¶§ ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3890621066093445 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9845558404922485 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10133533924818039 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.46462005376815796 'particle/f_score‚0.4080885648727417 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38006988167762756 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.46206894516944885 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123459696769714 _runtime‚11739.907292801 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14451533555984497 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.019396550953388214 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43183085322380066 ) Purity/3FRt1SRqqt1FBt/FRt_purity‚0.25 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.011437909677624702 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9911864399909973 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5488251447677612 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_5_of_7‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10909444838762283 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016875647008419037 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7355797290802002 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5031671524047852 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08662201464176178 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4834437966346741 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4173569083213806 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22836972773075104 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.47877761721611023 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6764705777168274 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5049228072166443 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1505754142999649 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18918919563293457 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 particle/accuracy_2_of_7‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9914783239364624 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_7‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.31274688243865967 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9910677671432495 jet/accuracy_6_of_6‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.513451874256134 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.520516574382782 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5603677034378052 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4480288028717041 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2021058201789856 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 6validation_average_jet_accuracy‚0.4215868413448334 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9733333587646484 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.015809010714292526 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14647655189037323 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3757380247116089 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.253352552652359 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 particle/accuracy_2_of_5‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_1_of_4‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4799100458621979 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.477710485458374 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.36666667461395264 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9512432217597961 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14626437425613403 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.021813547238707542 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4600282609462738 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.27õ¸ñdq3406751453876495 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9928160905838013 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5583333373069763 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09555555135011673 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.06578947603702545 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 _step‚993 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 particle/accuracy_7_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9886921644210815 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.013701708056032658 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002365616150200367 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24438293278217316 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1688058227300644 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9922755360603333 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.44948869943618774 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.49790191650390625 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 5 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.49038165807724 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12037035822868347 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16284695267677307 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00805696565657854 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2311508059501648 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4600282609462738 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9952381253242493 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2490740865468979 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7884282691550264e+09 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 particle/accuracy_3_of_4‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9904405474662781 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.48779600858688354 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05480317398905754 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5029312968254089 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.09935609996318817 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14022952318191528 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.005201044492423534 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20276951789855957 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.992125391960144 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4880274534225464 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3901515305042267 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5251725316047668 jet/accuracy_7_of_7‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23736511170864105 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 jet/accuracy_4_of_7‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.007258064113557339 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5309696793556213 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17730192840099335 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818182468414307 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42795565724372864 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9954648613929749 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9139065146446228 -jet/accuracy_4_of_5‚0.0070340498350560665 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13456284999847412 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.45900219678878784 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4476902484893799 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02202066034078598 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1513010710477829 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.14166666567325592 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4097001850605011 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.530308723449707 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3447779715061188 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1937241107225418 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03928571566939354 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10615752637386322 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6459094285964966 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17631317675113678 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909092128276825 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2518450915813446 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.46479716897010803 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5586805939674377 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3888888955116272 particle/accuracy_3_of_7‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3944444954395294 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_5‚0.4887993037700653 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39836594462394714 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1311294287443161 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 jet/accuracy_2_of_7‚0 jet/accuracy_6_of_7‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6842105388641357 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10823754966259003 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982219338417053 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 jet/accuracy_1_of_6‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5394098162651062 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4276094138622284 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08093836158514023 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17002823948860168 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_3_of_5‚1 -"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5625 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3860046863555908 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4493964910507202 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 *loss/val_total_loss‚41.214762346511144 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2716699242591858 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1643274873495102 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4561403691768646 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 'particle/accuracy‚0.853288471698761 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16997934877872467 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23116064071655273 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5155717730522156 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5239859223365784 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1736111044883728 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.32296502590179443 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13617977499961853 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 +jet/accuracy_3_of_3‚0.01917058788239956 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5472807884216309 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666269302368 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_6‚1 jet/accuracy_1_of_7‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.022440627217292786 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2961314618587494 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16145820915699005 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20459268987178802 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4022029936313629 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.494047611951828 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 +particle/sensitivity‚0.3240649402141571 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.40333330631256104 particle/accuracy_1_of_7‚1 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16145820915699005 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17875371873378754 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.027411900460720062 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21494251489639282 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 epoch‚9 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.822483241558075 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4954071342945099 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14453798532485962 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4734683632850647 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.011437909677624702 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.13333334028720856 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.44516077637672424 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9930745363235474 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9938411116600037 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38226962089538574 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.443513423204422 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13517972826957703 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.20580807328224182 +jet/accuracy_2_of_2‚0.10414063185453415 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0644872859120369 particle/accuracy_1_of_3‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975067377090454 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4046783149242401 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1967547982931137 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39263394474983215 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40275508165359497 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.007258064113557339 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 particle/accuracy_6_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0674603208899498 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4607204496860504 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.15625 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12764334678649902 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9855191111564636 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007407407276332378 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.042635660618543625 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968628287315369 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.15625 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06028369069099426 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5122374892234802 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.060452066361904144 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9930830597877502 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4523819088935852 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.71875 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.413194477558136 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12522676587104797 *jet/accuracy_2_of_4‚0.3583986163139343 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.05781489983201027 jet/accuracy_5_of_6‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.41929295659065247 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.24396973848342896 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.26897308230400085 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46655434370040894 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4369369149208069 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 trainer/global_step‚12089 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0072058457881212234 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5253464579582214 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.42548272013664246 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4127906858921051 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6755685210227966 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14526574313640594 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07381182163953781 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13755717873573303 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.13461539149284363 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.36970648169517517 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_6‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19862817227840424 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3433035910129547 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754689216614 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5666666626930237 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.026164688169956207 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 6 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.123795285820961 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.533699631690979 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2925925850868225 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013310089707374573 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_4_of_4‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9903714656829834 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12494172155857086 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_5_of_6‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5399700403213501 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5833231210708618 *jet/accuracy_1_of_3‚0.8529576063156128 *jet/accuracy_1_of_5‚0.9300691485404968 +jet/accuracy_2_of_6‚0.47435900568962097 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24237830936908722 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4406779408454895 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_3‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9994708895683289 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.7222222089767456 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4195401966571808 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.39945533871650696átéÐQ·- mb2gxnxyj? íÿäÔ¨ùœn0`Trainer.fit` stopped: `max_epochs=10` reached. œ—ùž¸- mb2gxnxyj‹ îÿäÔÀßú'zEpoch 9/9 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 1209/1209 0:18:18 • 0:00:00 1.10it/s v_num: xnxy-/$¹- mb2gxnxyj îÿäÔØž( __HŸ4º-/ _wandbruntime‚11742  _runtime‚11742Ž$7y»-‚(’Þ[